ass日本风韵熟妇pics男人扒开女人屁屁桶到爽|扒开胸露出奶头亲吻视频|邻居少妇的诱惑|人人妻在线播放|日日摸夜夜摸狠狠摸婷婷|制服 丝袜 人妻|激情熟妇中文字幕|看黄色欧美特一级|日本av人妻系列|高潮对白av,丰满岳妇乱熟妇之荡,日本丰满熟妇乱又伦,日韩欧美一区二区三区在线

森林生態(tài)站大數據快速存儲與索引方法
CSTR:
作者:
作者單位:

作者簡介:

通訊作者:

中圖分類號:

基金項目:

中央高?;究蒲袠I(yè)務費專項資金項目(BLX201923)和國家自然科學基金項目(32071775)


Fast Storage and Indexing Method of Big Data in Forest Ecological Station
Author:
Affiliation:

Fund Project:

  • 摘要
  • |
  • 圖/表
  • |
  • 訪問統(tǒng)計
  • |
  • 參考文獻
  • |
  • 相似文獻
  • |
  • 引證文獻
  • |
  • 資源附件
  • |
  • 文章評論
    摘要:

    針對森林生態(tài)站中大量圖像,、視頻、GIS數據等非結構化數據以及生態(tài)指標等結構化數據存儲效率低,、檢索性能差的問題,,提出了基于Hadoop和HBase的森林生態(tài)站大數據存儲框架?;谒岢龅目蚣?,給出了森林生態(tài)數據存儲業(yè)務流程,并對森林生態(tài)大數據平臺涉及的核心技術進行了優(yōu)化:①設計預分區(qū)算法保證數據在集群中均勻分布,。②根據生態(tài)數據特點科學設計了RowKey,,實現(xiàn)生態(tài)數據的快速檢索,。③針對原生HBase不支持多條件查詢問題,,設計基于索引數據和服務器性能評估的ElasticSearch索引分片放置策略,以此基于ElasticSearch的二級非主鍵索引技術優(yōu)化多條件檢索HBase生態(tài)數據庫,。④針對生態(tài)站海量小圖像存儲困難問題,,提出基于數據站點及時間關聯(lián)性的打包合并策略。⑤解析GIS數據使之進行高效存儲,。通過實驗對以上理論進行驗證,。結果表明,ElasticSearch索引分片放置策略比默認分片策略的查詢時間平均減少20 ms,,比基于改變ElasticSearch評分策略的查詢時間平均減少20 ms,。結構化數據規(guī)模為1×108條時,,系統(tǒng)的檢索時間為1.045 s,比原生HBase檢索速度提升3.99倍,,在非結構化數據為1×107條時,,采用數據站點及時間關聯(lián)性的打包小圖像策略是基于SequenceFile合并效率的1.15倍,是原生HBase的1.79倍;在1×104次并發(fā)用戶的情況下,,優(yōu)化后的每秒查詢數是原來的1.88倍,,每秒吞吐量是優(yōu)化前的1.74倍,系統(tǒng)響應時間比優(yōu)化前降低69.5%,。結果表明,,本文所提出的方案在集群負載均衡、海量結構化和非結構化數據檢索效率以及系統(tǒng)吞吐量等方面都有了明顯的性能提升,,為森林生態(tài)數據的存儲和管理提供了必要的理論基礎和技術實現(xiàn),。

    Abstract:

    Aiming at the problems of low storage efficiency and poor retrieval performance of a large number of unstructured data such as images, videos, GIS data and ecological indicators in the forest ecological station, a forest ecological station big data storage framework was proposed based on Hadoop and HBase. Based on the proposed framework, the business process of forest ecological data storage was given and the core technologies involved in the forest ecological big data platform was optimized.A pre-partitioning algorithm was designed to ensure that the data was evenly distributed in the cluster. According to the characteristics of ecological data, the RowKey was scientifically designed to achieve rapid retrieval of ecological data. Aiming at the problem that native HBase did not support multi-condition query, an ElasticSearch index shard placement strategy was designed based on index data and server performance evaluation, and the multi-condition search HBase ecological database was optimized based on ElasticSearch's secondary non-primary key index technology. In view of the difficulty of storing large amounts of small pictures in the ecological station, a package and merge strategy was proposed based on data sites and time relevance. GIS data was analyzed for efficient storage. The above theory was verified through experiments. The results showed that the ElasticSearch index shard placement strategy reduced the query time by an average of 20 ms compared with the default shard strategy. The average query time was reduced by 20 ms compared with that based on changing the ElasticSearch scoring policy. When the structured data size was 1×108, the retrieval time of the system was 1.045 s, which was 3.99 times faster than the native HBase retrieval, and when the unstructured data was 1×107 pieces, the based on data site and time correlation package small picture strategy was 1.15 times that of SequenceFile-based merging efficiency and 1.79 times that of native HBase.In the case of 1×104 concurrent users, after optimization, the number of queries per second was 1.88 times as much as before, the throughput per second was 1.74 times as much as before, and the system response time was 69.5% lower than that before optimization. From the above results, it can be seen that the solution proposed had significant performance improvements in cluster load balancing, massive structured and unstructured data retrieval efficiency, and system throughput, which provided the necessary theoretical foundation and technical realization for the storage and management of forest ecological data.

    參考文獻
    相似文獻
    引證文獻
引用本文

王新陽,賈相宇,陳志泊,崔曉暉,許福.森林生態(tài)站大數據快速存儲與索引方法[J].農業(yè)機械學報,2021,52(8):195-204,212. WANG Xinyang, JIA Xiangyu, CHEN Zhibo, CUI Xiaohui, XU Fu. Fast Storage and Indexing Method of Big Data in Forest Ecological Station[J]. Transactions of the Chinese Society for Agricultural Machinery,2021,52(8):195-204,,212.

復制
分享
文章指標
  • 點擊次數:
  • 下載次數:
  • HTML閱讀次數:
  • 引用次數:
歷史
  • 收稿日期:2021-02-08
  • 最后修改日期:
  • 錄用日期:
  • 在線發(fā)布日期: 2021-08-10
  • 出版日期:
文章二維碼