Glog: General Log
Glog is a high-performance, cross-platform mobile logging component. Use mmap I/O, support synchronous/asynchronous write mode, format log with custom binary format, SDK user can choose whatever mechanism you like(eg. JSON, Protobuf). The underlaying file format is flexible, high-performance and fault-tolerant. Glog support log compression, encryption and cleanup, also includes reading module written by C++
I. WHY CHOOSE
Most of the logging component takes over the data serialization process, convert data object to binary data by JSON or other mechanism, then write binary data to file. The advantage of this is easy to use, but lack of flexible. Glog decouple the serialization process with underlaying log, remain high performance and reliable.
Technical implementation
- mmap I/O
- Customized binary file format, store each log with binary data, upper layer is responsible for serialization(Protobuf is recommended).Add sync mark to the end of each log to support fault-tolerant.
- Stream compression to reduce CPU usage
- Encrypt each log by AES CFB-128 and ECDH
- Support synchronous/asynchronous write mode, asynchronous mode implemented by message queue written by C++
- Support incremental archive(per day) and full archive(per file)
- Build-in read module
Benchmark
Phone model | Log SDK | 10K write | 100K write |
---|---|---|---|
Samsung Galaxy S10+ Android 11 | glog | 21 ms | 182 ms |
glog+pb | 54 ms | 335 ms | |
xlog | 207 ms | 1961 ms | |
logan | 250 ms | 6469 ms | |
Huawei Honor Magic 2 Android 10 | glog | 38 ms | 286 ms |
glog+pb | 84 ms | 505 ms | |
xlog | 263 ms | 2165 ms | |
logan | 242 ms | 3643 ms | |
Xiaomi 10 Android 11 | glog | 27 ms | 244 ms |
xlog | 198 ms | 1863 ms | |
logan | 210 ms | 4238 ms | |
Huawei Mate 40 pro HarmonyOS 2.0.0 | glog | 30 ms | 257 ms |
xlog | 275 ms | 2484 ms | |
logan | 260 ms | 4020 ms | |
OPPO R11 Android 8.1.0 | glog | 63 ms | 324 ms |
glog+pb | 234 ms | 1611 ms | |
xlog | 464 ms | 3625 ms | |
logan | 430 ms | 5355 ms | |
iPhone 12 128G iOS 14.8 | glog | 7 ms | 29 ms |
xlog | 152 ms | 1079 ms | |
logan | 162 ms | 12821 ms | |
iPhone 8 64G iOS 13.7 | glog | 12 ms | 50 ms |
xlog | 242 ms | 2106 ms | |
logan | 251 ms | 38312 ms |
II. HOW TO USE
Android
IMPORT
The latest release is available on Maven Central.
allprojects {
repositories {
mavenCentral()
}
}
dependencies {
implementation "cn.huolala.glog.android:glog-android-static:1.0.0"
// implementation "cn.huolala.glog.android:glog-android-shared:1.0.0"
}
QUIC START
public class SampleApp extends Application {
private Glog glog;
@Override
public void onCreate() {
super.onCreate();
// Global initialization, setup internal log level for debug
Glog.initialize(BuildConfig.DEBUG ? InternalLogLevelDebug : InternalLogLevelInfo);
setup();
}
// initialize instance
private void setup() {
glog = new Glog.Builder(getApplicationContext())
.protoName("glog_identify") // instance identity, instance with same identity will create only once
.encryptMode(Glog.EncryptMode.AES) // Encryption mode
.key("") // ECDH Server public key
.incrementalArchive(true) // Incremental archive,log within a day write to same file
.build();
}
// write log
private void write() {
byte[] data = serialize(); // serizlize data
glog.write(data); // write binary data
}
// read log
private void read() {
ArrayList<String> logFiles = new ArrayList<>();
glog.getArchiveSnapshot(logFiles, 10, 100 * 1024); // get snapshot of log files list, flush cache if log num >= 10 or log size >= 100KB in cache
byte[] inBuffer = new byte[Glog.getSingleLogMaxLength()];
for (String logFile : logFiles) {
try (Glog.Reader reader = glog.openReader(logFile)) {
while (true) {
int n = reader.read(inBuffer);
if (n < 0) {
break;
} else if (n == 0) { // fault-tolerant
continue;
}
byte[] outBuffer = new byte[n];
System.arraycopy(inBuffer, 0, outBuffer, 0, n);
deserialize(outBuffer); // deserialize binary data
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
iOS
IMPORT
pod 'Huolala-Glog'
QUIC START
// initialize default instance
Glog *glog = [Glog defaultGlog];
// write data
glog.write(data);
// get snapshot of log files list, flush cache if log num >= 10 or log size >= 100KB in cache
NSArray *logFiles = [self.glog getArchiveSnapshot:true minLogNum:10 totalLogSize:100 * 1024];
NSMutableArray *dataArr = [NSMutableArray new];
for (NSString *item in logFiles) {
GlogReader *reader = [self.glog openReader:item];
NSInteger length = 0;
NSData *data;
do {
length = [reader read:16 * 1024 data:&data];
if (length == 0) { // fault-tolerant
continue;
} else if (length < 0){
break;
}
if (data) {
NSString *dataString = [[NSString alloc] initWithData:data encoding:NSASCIIStringEncoding];
[dataArr addObject:dataString];
}
} while (YES);
NSLog(@"all the data: %@",dataArr);
[self.glog closeReader:reader];
}
III. TOOLS
- Misc/Reader
- Misc/generate-ecc-key.py
IV. AUTHOR
V. LICENSE
Copyright 2022 Huolala, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.