BigTIFF version of libtiff library

The library retains the capability of writing standard TIFF files, compatible with previous versions of the library and other existing software. Standard TIFF files contain 32-bit offsets to directories and image data. This limits the total size of a standard TIFF file to 4GB (2^32).

The library now has the capability of writing BigTIFF files, which contain 64-bit offsets to directories and image data. For a BigTIFF file there is no practical limit to the size of a file.

There are several places where the format of a BigTIFF file differs from a standard TIFF file:

TIFF file header

TIFF directory format

For some image data the data themselves consist of file offsets, such as tile or strip data locations. In a standard TIFF file these offsets are 32-bit, with 16-bit offsets also supported for backward compatibility. In a BigTIFF file these offsets may be 64-bit values.

Two new data types have been defined for 64-bit integers: TIFF_LONG8 and TIFF_SLONG8. A new data type has also been defined for 64-bit IFD offsets: TIFF_IFD8.

When a file is opened for reading, the TIFF version in the file header is used to determine whether the file is a standard TIFF file, which uses 32-bit offsets, or a BigTIFF file, which uses 64-bit offsets. In either case, 64-bit offsets are used in internal data structures.

When a file is written, it is initially unknown whether it will have to become a BigTIFF file. If the total size is less than 2^32, there is no reason to make it a BigTIFF file. The library begins by writing a standard TIFF file with 32-bit offsets. It leaves 8 extra bytes of space after the file header in case the file will need to become a BigTIFF file, but otherwise the processing is identical to before. Internally, the library uses 64-bit offsets, but externally the file contains 32-bit offsets.

Image data are written to the file as before, with no change. All internal data offsets are 64-bit and all I/O routines have been modified to use 64-bit file offsets; files larger than 2^32 bytes are supported seamlessly.

When a directory is written, if the file is not already in BigTIFF format, note is made whether the directory will be located beyond 2^32 from the start of the file. If so, the file is converted into a BigTIFF file.

The following things are done to convert from standard TIFF to BigTIFF at this point:

• A new file header is written with the BigTIFF format.
• Each of the directories which have already been written into the file are rewritten as BigTIFF directories. A new chain of directories is established beginning with a 64-bit offset in the file header. Image data are left in place, only the directories are rewritten. The old 32-bit directories are left as dead space in the file.
• If a directory contains subdirectories, the subdirectory offsets are updated with 64-bit values to point to the rewritten directories.

After this point, all directories are written with the BigTIFF format, with 64-bit data offsets, 64-bit strip/tile offsets, and related values. Subdirectory offsets are written as 64-bit values, and the pointers linking each directory to the next use 64-bit offsets.

The general technique was to leave all data structures and types alone with the exception of offsets within the file. All file offsets are stored internally and processed in the TIFF objects as toff_t, an unsigned 64-bit value.

The library was originally designed to keep tables with file block offsets and lengths in memory. A subsequent update modified this logic to manage block tables in segments. Only fixed portions of the block tables are kept in memory, paged in and out from the TIFF file on disk. This enables files of any size to be processed.

The majority of the API to the TIFF object remained unchanged. The main change is that the toff_t data type used by some API functions for file offsets changed from a 32-bit integer to a 64-bit integer. Applications which use API functions involving file offsets require recompilation and may require minor changes to handle 64-bit values.

The TIFFTAG_SUBIFD tag formerly used 32-bit values and now uses 64-bit values for subdirectory offsets. Applications which use TIFFTAG_SUBIFD require recompilation and may require minor changes to handle 64-bit offsets.

The TIFFOpen function supports two new optional modes:

• '4': force reading and writing standard TIFF files. On input, if the file is BigTIFF, an error is reported. On output, if the file reaches 2^32 in size, an error is reported.
• '8': force reading and writing BigTIFF files. On input, if the file is standard TIFF, an error is reported. On output, the file will be forced to use the BigTIFF format regardless of size.

A new TIFFIsBigTIFF function returns whether the current file is a BigTIFF.

TIFFOpen mode '4'  // force reading and writing standard TIFF files
TIFFOpen mode '8'  // force reading and writing BigTIFF files
TIFFIsBigTIFF()    // return whether the current file is BigTIFF

The source code has been modified and tested on Win32 using MS and Intel compilers, on Linux using RedHat, and on Mac OS X. Other platform-specific code has not been modified or tested.

The tiffinfo tool provided with libtiff was updated to display file information for both standard TIFF and BigTIFF files. The other tools provided with libtiff were not tested or updated.

HEADERS
---- 4.0 ----
tiff.h
  - define int64 and uint64 data type
  - redefine TIFFheader as union of standard and BigTIFF formats
  - redefine TIFFDirEntry as union of standard and BigTIFF forms
  - define TIFF_LONG8, TIFF_SLONG8, TIFF_IFD8 64-bit data types

tiffconf.h
  - HAVE_INT64 macro (use "__int64" if set, else "long long")

tiffio.h
  - redefine toff_t data type
  - define new API functions:
      TIFFIsBigTIFF
      TIFFSwabLongLong
      TIFFSwabArrayOfLongLong

tiffiop.h
  - eliminate typeshift and typemask
  - define new tif_flags:
      TIFF_ISBIGTIFF  this is a BigTIFF file
      TIFF_NOBIGTIFF  do not process BigTIFF files (error instead)
  - define new macros:
      isBigTIFF       whether this is a BigTIFF file
      noBigTIFF       whether BigTIFF files are prevented
      isBigOff        whether offset is > 2^32
  - define macros to access directory offset in header
      TIFFGetHdrDirOff, TIFFSetHdrDirOff
  - define macros to access directory counts in IFDs
      TIFFGetDirCnt, TIFFSetDirCnt, TIFFSwabDirCnt, TIFFDirCntLen
  - define macros to access directory offsets for IFDs
      TIFFGetDirOff, TIFFSetDirOff, TIFFSwabDirOff, TIFFDirOffLen
  - define macros to access directory entry data
      TDIREntryLen, TDIREntryNext
      TDIRGetEntryCount, TDIRSetEntryCount, TDIRSwabEntryCount
      TDIRAddrEntryOff, TDIRGetEntryOff, TDIRSetEntryOff,
      TDIRSwabEntryOff, TDIREntryOffLen
  - add prototype for _TIFFsetLong8Array

tifvers.h
  - update version to 4.0.0 (20070401)

tif_dir.h
  - remove TIFFInsertData and TIFFExtractData macros

---- 4.1 ----
tiffiop.h
  - add prototypes for _TIFFGetOffset, _TIFFGetByteCount, _TIFFSetOffset,
    _TIFFFlushOffsets(TIFF*), _TIFFSetByteCount, _TIFFFlushByteCounts

tifvers.h
  - update version to 4.1.0 (20111201)

tif_dir.h
  - define STRIPBUFMAX macro for size of offsets and byte counts arrays
  - define new td_strip... data in TIFFDirectory for offsets and byte counts arrays

CODE
---- 4.0 ----
tif_dir.c
  - implement _TIFFsetLong8Array
  - add 32/64-bit logic for TIFFTAG_SUBIFD
  - update TIFFAdvanceDirectory, TIFFNumberOfDirectories, TIFFSetDirectory,
    and TIFFUnlinkDirectory to use macros to access directory chain
  - implement TIFFSetSubDirectoryB and TIFFCurrentDirOffsetB functions

tif_dirinfo.c
  - add TIFF_LONG8 variation for TIFFTAG_STRIPOFFSETS
  - add TIFF_LONG8 variation for TIFFTAG_STRIPBYTECOUNTS
  - add TIFF_LONG8 and TIFF_IFD8 variations for TIFFTAG_SUBIFD
  - update TIFFDataWidth and _TIFFDataSize to support 64-bit types

tif_dirread.c
  - update TIFFReadDirectory to use macros to access header fields and directory entries
  - add 32-bit/64-bit logic for TIFFTAG_SUBIFD
  - replace TIFFExtractData with TIFFFetch<x>Array calls
  - update TIFFReadCustomDirectory to use macros to access header fields and directory entries
  - replace TIFFFetchStripThing with TIFFFetchByteCounts and TIFFFetchOffsets
  - implement TIFFFetchLong8Array
  - support 64-bit data types in TIFFFetchData
  - update TIFFFetch<x>Array functions to support 64-bit data in BigTIFF entries

tif_dirwrite.c
  - update _TIFFWriteDirectory to use macros to access header fields and directory entries
  - support 32-bit or 64-bit strip/tile offsets
  - replace TIFFInsertData with TIFFWrite<x>Array calls
  - implement TIFFWriteLong8Array
  - support 64-bit data types in TIFFWriteData
  - update TIFFWrite<x>Array functions to support 64-bit data in BigTIFF entries
  - update TIFFRewriteDirectory to use macros to access directory chain
  - update TIFFLinkDirectory to use macros to access directory chain
  - implement local TIFFMakeBigTIFF function (convert directories, etc.)

tif_ojpeg.c
  - use macro for directory entry length

tif_open.c
  - eliminate use of typemask and typeshift tables
  - update TIFFClientOpen to support BigTIFF header
  - support '4' mode to explicitly prevent reading and writing BigTIFF files
  - support '8' mode to explicitly force reading and writing BigTIFF files
  - read/write standard or BigTIFF header as required
  - implement TIFFIsBigTIFF

tif_print.c
  - replace cascading if with switch
  - support 64-bit data types
  - support 64-bit directory and file offsets
  - format strip/tile offset array as 64-bit toff_t
  - format SUBIFD array as 64-bit toff_t

tif_swab.c
  - add TIFFSwabLong8 routine
  - add TIFFSwabArrayOfLong8 routine

tif_unix.c
  - modify Unix API calls to use 64-bit offsets everywhere

tif_win32.c
  - modify Windows API calls to use 64-bit offsets everywhere

tif_write.c
  - modify TIFFSetupStrips and TIFFGrowStrips for 64-bit offsets

---- 4.1 ----
tif_dir.c
  - initialize new td_strip... data in TIFFDirectory from strip/tile tags

tif_dirread.c
  - setup td_strip... data during tag processing
  - only check for sorted offsets and byte counts if not read only
  - use _TIFFGetOffset and _TIFFGetByteCount instead of accessing arrays
  - change EstimateStripByteCounts for new td_strip... data
  - implement _TIFFGetOffset and _TIFFSetOffset

tif_dirwrite.c
  - call _TIFFFlushOffsets and _TIFFFlushByteCounts prior to writing directory
  - set strip/tile tags from new td_strip... data
  - implement _TIFFSetOffset, _TIFFFlushOffsets, _TIFFSetByteCount, _TIFFFlushByteCounts

tif_ojpeg.c
  - use _TIFFGetOffset, _TIFFGetByteCount

tif_print.c
  - use _TIFFGetOffset, _TIFFGetByteCount

tif_read.c
  - use _TIFFGetOffset, _TIFFGetByteCount

tif_strip.c
  - use _TIFFGetOffset, _TIFFGetByteCount

tif_write.c
  - use _TIFFGetOffset, _TIFFGetByteCount, _TIFFSetOffset, _TIFFSetByteCount
  - modify TIFFSetupStrips to defer initialization to _TIFFGetOffset and _TIFFGetByteCount
  - check offsets and byte counts array locations when growing strip/tile