fix(ut,orc): fix orc read ut under debian and adjust some options

[SGZW]

Purpose

Linked issue: close #xxx

1. src/paimon/format/orc/orc_file_batch_reader.cpp

refer: https://github.com/apache/arrow/pull/34591/files,  
// Orc timestamp type is error-prone since it serializes values in the writer timezone
// and reads them back in the reader timezone. To avoid this, both the Apache Orc C++
// writer and reader set the timezone to GMT by default to avoid any conversion.
// We follow the same practice here explicitly to make sure readers are aware of this.

2. rest of fix:

// refer: https://github.com/eggert/tz/blob/main/asia#L653
// When using the Asia/Shanghai timezone under Debian, timestamps prior to 1901 have an
// additional offset of 5 minutes and 43 seconds

The specific verification steps are as follows: I modified the code related to ORC format and added some debug logs, focusing primarily on the code for timestamp time zone conversion. The details are as follows:

  void TimestampColumnReader::next(ColumnVectorBatch& rowBatch, uint64_t numValues, char* notNull) {
    ColumnReader::next(rowBatch, numValues, notNull);
    notNull = rowBatch.hasNulls ? rowBatch.notNull.data() : nullptr;
    TimestampVectorBatch& timestampBatch = dynamic_cast<TimestampVectorBatch&>(rowBatch);
    int64_t* secsBuffer = timestampBatch.data.data();
    secondsRle_->next(secsBuffer, numValues, notNull);
    int64_t* nanoBuffer = timestampBatch.nanoseconds.data();
    nanoRle_->next(nanoBuffer, numValues, notNull);

    // Construct the values
    for (uint64_t i = 0; i < numValues; i++) {
      if (notNull == nullptr || notNull[i]) {
        uint64_t zeros = nanoBuffer[i] & 0x7;
        nanoBuffer[i] >>= 3;
        if (zeros != 0) {
          for (uint64_t j = 0; j <= zeros; ++j) {
            nanoBuffer[i] *= 10;
          }
        }

        // ORC-306: compensate -1s for JDK bug in java.sql.Timestamp
        int64_t writerTime = secsBuffer[i] + epochOffset_;
        if (writerTime < 0 && nanoBuffer[i] > 999999) {
            writerTime -= 1;
        }
        if (!sameTimezone_) {
          std::stringstream s1,s2;
          writerTimezone_->print(s1);
          readerTimezone_->print(s2);
          std::cerr << "### writer zone ### \n" << s1.str() << std::endl;
          std::cerr << "### reader zone ### \n" << s2.str() << std::endl;
          // adjust timestamp value to same wall clock time if writer and reader
          // time zones have different rules, which is required for Apache Orc.
          const auto& wv = writerTimezone_->getVariant(writerTime);
          const auto& rv = readerTimezone_->getVariant(writerTime);
          std::cerr << "wv: " << wv.toString() << ", rv: " << rv.toString() << std::endl;
          if (!wv.hasSameTzRule(rv)) {
            // If the timezone adjustment moves the millis across a DST boundary,
            // we need to reevaluate the offsets.
            int64_t adjustedTime = writerTime + wv.gmtOffset - rv.gmtOffset;
            const auto& adjustedReader = readerTimezone_->getVariant(adjustedTime);
            writerTime = writerTime + wv.gmtOffset - adjustedReader.gmtOffset;
          }
        }
        std::cerr << "epochOffset_: " << epochOffset_ << std::endl;
        std::cerr << "secsBuffer[i]: " << secsBuffer[i] << ", writerTime: " << writerTime << std::endl;
        secsBuffer[i] = writerTime;
      }
    }
  }

In the Debian environment, I used Ubuntu 24.04's tzdata (wget http://security.ubuntu.com/ubuntu/pool/main/t/tzdata/tzdata_2025b-0ubuntu0.24.04.1_all.deb) and Debian's tzdata for the TZDIR environment variable separately, with the results as follows.

Debian:

### writer zone ###
Timezone file: /usr/share/zoneinfo/Asia/Shanghai
  Version: 2
  Future rule: CST-8
  standard CST 28800
  Variant 0: LMT 29143
  Variant 1: CDT 32400 (dst)
  Variant 2: CST 28800
  Transition: null (-576460752303423488) -> LMT
  Transition: 1900-12-31 15:54:17 (-2177481943) -> CST
  Transition: 1919-04-12 16:00:00 (-1600675200) -> CDT
  Transition: 1919-09-30 15:00:00 (-1585904400) -> CST
  Transition: 1940-05-31 16:00:00 (-933667200) -> CDT
  Transition: 1940-10-12 15:00:00 (-922093200) -> CST
  Transition: 1941-03-14 16:00:00 (-908870400) -> CDT
  Transition: 1941-11-01 15:00:00 (-888829200) -> CST
  Transition: 1942-01-30 16:00:00 (-881049600) -> CDT
  Transition: 1945-09-01 15:00:00 (-767869200) -> CST
  Transition: 1946-05-14 16:00:00 (-745833600) -> CDT
  Transition: 1946-09-30 15:00:00 (-733827600) -> CST
  Transition: 1947-04-14 16:00:00 (-716889600) -> CDT
  Transition: 1947-10-31 15:00:00 (-699613200) -> CST
  Transition: 1948-04-30 16:00:00 (-683884800) -> CDT
  Transition: 1948-09-30 15:00:00 (-670669200) -> CST
  Transition: 1949-04-30 16:00:00 (-652348800) -> CDT
  Transition: 1949-05-27 15:00:00 (-650019600) -> CST
  Transition: 1986-05-03 18:00:00 (515527200) -> CDT
  Transition: 1986-09-13 17:00:00 (527014800) -> CST
  Transition: 1987-04-11 18:00:00 (545162400) -> CDT
  Transition: 1987-09-12 17:00:00 (558464400) -> CST
  Transition: 1988-04-16 18:00:00 (577216800) -> CDT
  Transition: 1988-09-10 17:00:00 (589914000) -> CST
  Transition: 1989-04-15 18:00:00 (608666400) -> CDT
  Transition: 1989-09-16 17:00:00 (621968400) -> CST
  Transition: 1990-04-14 18:00:00 (640116000) -> CDT
  Transition: 1990-09-15 17:00:00 (653418000) -> CST
  Transition: 1991-04-13 18:00:00 (671565600) -> CDT
  Transition: 1991-09-14 17:00:00 (684867600) -> CST

### reader zone ###
Timezone file: /usr/share/zoneinfo/GMT
  Version: 2
  Future rule: GMT0
  standard GMT 0
  Variant 0: GMT 0
  Transition: null (-576460752303423488) -> GMT

wv: LMT 29143, rv: GMT 0
epochOffset_: 1420041600
secsBuffer[i]: -3660591639, writerTime: -2240520897

Ubuntu 24.04

### writer zone ###
Timezone file: /home/zhangwei.95/zoneinfo/usr/share/zoneinfo//Asia/Shanghai
  Version: 2
  Future rule: CST-8
  standard CST 28800
  Variant 0: LMT 29143
  Variant 1: CDT 32400 (dst)
  Variant 2: CST 28800
  Transition: 1900-12-31 15:54:17 (-2177481943) -> CST
  Transition: 1919-04-12 16:00:00 (-1600675200) -> CDT
  Transition: 1919-09-30 15:00:00 (-1585904400) -> CST
  Transition: 1940-05-31 16:00:00 (-933667200) -> CDT
  Transition: 1940-10-12 15:00:00 (-922093200) -> CST
  Transition: 1941-03-14 16:00:00 (-908870400) -> CDT
  Transition: 1941-11-01 15:00:00 (-888829200) -> CST
  Transition: 1942-01-30 16:00:00 (-881049600) -> CDT
  Transition: 1945-09-01 15:00:00 (-767869200) -> CST
  Transition: 1946-05-14 16:00:00 (-745833600) -> CDT
  Transition: 1946-09-30 15:00:00 (-733827600) -> CST
  Transition: 1947-04-14 16:00:00 (-716889600) -> CDT
  Transition: 1947-10-31 15:00:00 (-699613200) -> CST
  Transition: 1948-04-30 16:00:00 (-683884800) -> CDT
  Transition: 1948-09-30 15:00:00 (-670669200) -> CST
  Transition: 1949-04-30 16:00:00 (-652348800) -> CDT
  Transition: 1949-05-27 15:00:00 (-650019600) -> CST
  Transition: 1986-05-03 18:00:00 (515527200) -> CDT
  Transition: 1986-09-13 17:00:00 (527014800) -> CST
  Transition: 1987-04-11 18:00:00 (545162400) -> CDT
  Transition: 1987-09-12 17:00:00 (558464400) -> CST
  Transition: 1988-04-16 18:00:00 (577216800) -> CDT
  Transition: 1988-09-10 17:00:00 (589914000) -> CST
  Transition: 1989-04-15 18:00:00 (608666400) -> CDT
  Transition: 1989-09-16 17:00:00 (621968400) -> CST
  Transition: 1990-04-14 18:00:00 (640116000) -> CDT
  Transition: 1990-09-15 17:00:00 (653418000) -> CST
  Transition: 1991-04-13 18:00:00 (671565600) -> CDT
  Transition: 1991-09-14 17:00:00 (684867600) -> CST

### reader zone ###
Timezone file: /home/zhangwei.95/zoneinfo/usr/share/zoneinfo//GMT
  Version: 2
  Future rule: GMT0
  standard GMT 0
  Variant 0: GMT 0

wv: CST 28800, rv: GMT 0
epochOffset_: 1420041600
secsBuffer[i]: -3660591639, writerTime: -2240521240

The result is self-evident: Debian’s tzdata contains an extra LMT Timezone entry, which ultimately leads to a 5 minutes and 30 seconds offset in the final result. For the time being, I have bypassed this issue by detecting the OS version. To resolve the problem at its root, we need to ensure that the writer zone of the ORC file is set to the UTC/GMT time zone for data writing, thus avoiding similar issues.

Tests

API and Format

Documentation

[SGZW]

@lucasfang @lxy-9602 @ChaomingZhangCN PTAL, thanks!