博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
Offline Edits Viewer Guide
阅读量:4155 次
发布时间:2019-05-25

本文共 2463 字,大约阅读时间需要 8 分钟。

Offline Edits Viewer Guide

Overview

Offline Edits Viewer is a tool to parse the Edits log file. The current processors are mostly useful for conversion between different formats, including XML which is human readable and easier to edit than native binary format.

The tool can parse the edits formats -18 (roughly Hadoop 0.19) and later. The tool operates on files only, it does not need Hadoop cluster to be running.

Input formats supported:

  1. binary: native binary format that Hadoop uses internally
  2. xml: XML format, as produced by xml processor, used if filename has .xml (case insensitive) extension

The Offline Edits Viewer provides several output processors (unless stated otherwise the output of the processor can be converted back to original edits file):

  1. binary: native binary format that Hadoop uses internally
  2. xml: XML format
  3. stats: prints out statistics, this cannot be converted back to Edits file

Usage

bash$ bin/hdfs oev -i edits -o edits.xml
Flag Description
[-i ; --inputFileinput file Specify the input edits log file to process. Xml (case insensitive) extension means XML format otherwise binary format is assumed. Required.
[-o ; --outputFile]output file Specify the output filename, if the specified output processor generates one. If the specified file already exists, it is silently overwritten. Required.
[-p ; --processor]processor Specify the image processor to apply against the image file. Currently valid options are binaryxml (default) and stats.
[-v ; --verbose] Print the input and output filenames and pipe output of processor to console as well as specified file. On extremely large files, this may increase processing time by an order of magnitude.
[-h ; --help] Display the tool usage and help information and exit.

Case study: Hadoop cluster recovery

In case there is some problem with hadoop cluster and the edits file is corrupted it is possible to save at least part of the edits file that is correct. This can be done by converting the binary edits to XML, edit it manually and then convert it back to binary. The most common problem is that the edits file is missing the closing record (record that has opCode -1). This should be recognized by the tool and the XML format should be properly closed.

If there is no closing record in the XML file you can add one after last correct record. Anything after the record with opCode -1 is ignored.

Example of a closing record (with opCode -1):

-1

转载地址:http://rtqti.baihongyu.com/

你可能感兴趣的文章
numpy数组展开方法
查看>>
python同时随机排序多个数组
查看>>
Xgboost自定义目标函数
查看>>
多种方法判断某个值是否在一个有序的二维数组中
查看>>
TypeError: cannot perform std with type Sparse[float64, 0.0]
查看>>
常见的类别型数据列编码方法汇总
查看>>
牛顿法和最优化
查看>>
特征金字塔网络总结
查看>>
修改pytorch和Keras预训练模型路径
查看>>
yolov4中的route和shortcut层
查看>>
Pytorch中loss计算解析
查看>>
BF16格式数据
查看>>
TypeError: ufunc ‘true_divide‘ output (typecode ‘d‘) could not be coerced to provided outp
查看>>
记录Ubuntu18.04-cuda10.1-opencv4配置caffe过程
查看>>
tensorflow2.2中定义的ResNet和ResneXt中的bottleneck结构
查看>>
Pytorch中nn.Conv2d的dilation
查看>>
onnx2caffe:KeyError: ‘broadcast‘
查看>>
感受野知识点总结
查看>>
GoogleNet总结
查看>>
pandas学习笔记-------时间戳转日期时间型数据
查看>>