Error message when decompressing .vcz data with vcztools

[browning-lab]

I am testing vcztools on a 1 kb region of the phased UKB 500k WGS data (chr10:10000000-10001000).

The data were compressed with the commands:

vcf2zarr explode -Q test.vcf.gz  test.icf
vcf2zarr encode  -Q test.icf     test.vcz

Data decompression with the command:

vcztools view test.vcz > test1.vcf

failed and produced the error message: "Error: Error occured [sic]: -204"

[jeromekelleher]

Thanks for the report @browning-lab! I'm a bit baffled by how we get the error message, but it looks like the problem is something to do with a weird field. Error -204 is VCZ_ERR_FIELD_UNSUPPORTED_NUM_COLUMNS which only gets thrown in one place:

vcztools/lib/vcf_encoder.c

Line 493 in 475d7c9

ret = VCZ_ERR_FIELD_UNSUPPORTED_NUM_COLUMNS;

Which says the number of columns in the field is 0 or less. Would you mind putting in a bit more information here so we can debug this and handling it better for other UKB users?

The output of

vcf2zarr inspect test.vcz

would be a great start.

[browning-lab]

Here is the code in my shell script, running on the UKB RAP: time ( vcf2zarr explode -Q "${in_file}" "${icf_file}" vcf2zarr encode -Q "${icf_file}" "${zarr_dir}" ) echo "Before inspect" 1>&2 vcf2zarr inspect ${zarr_dir} echo "After inspect" 1>&2 And here is the combined stdout + stderr output from the preceding commands: 2025-12-15 19:27:46 job-J505z08JzkJ7vx7b58GjZKV1 bio2zarr.vcz[5214] WARNING Field call_FT is a string; max memory usage may be a significant underestimate 2025-12-15 19:27:47 job-J505z08JzkJ7vx7b58GjZKV1 bio2zarr.vcz[5214] WARNING Field call_LAA is a string; max memory usage may be a significant underestimate 2025-12-15 19:27:47 job-J505z08JzkJ7vx7b58GjZKV1 bio2zarr.vcz[5214] WARNING Field call_FT is a string; max memory usage may be a significant underestimate 2025-12-15 19:27:47 job-J505z08JzkJ7vx7b58GjZKV1 bio2zarr.vcz[5214] WARNING Field call_LAA is a string; max memory usage may be a significant underestimate real 0m37.953s user 0m34.741s sys 0m3.312s Before inspect Traceback (most recent call last): File "/usr/local/bin/vcf2zarr", line 8, in <module> sys.exit(vcf2zarr_main()) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1485, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1406, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1873, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1269, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 824, in invoke return callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/bio2zarr/cli.py", line 335, in inspect data = vcf_mod.inspect(path) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/bio2zarr/vcf.py", line 1589, in inspect return obj.summary_table() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/bio2zarr/vcz.py", line 1128, in summary_table "chunk_size": core.display_size(array.nbytes / array.nchunks), ~~~~~~~~~~~~~^~~~~~~~~~~~~~~

…

On Mon, Dec 15, 2025 at 1:29 AM Jerome Kelleher ***@***.***> wrote: *jeromekelleher* left a comment (sgkit-dev/vcztools#245) <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/issues/245*issuecomment-3654629813__;Iw!!K-Hz7m0Vt54!hnoAgJ_PVqTf-CnEC-enLmBxisRuCVsSwkbZddIbSc1fWHlGRp7MU80KhB0skZby1yBEk6H5xf0AbA9ok1z_3s4hsbZ07_U$> Thanks for the report @browning-lab <https://urldefense.com/v3/__https://github.com/browning-lab__;!!K-Hz7m0Vt54!hnoAgJ_PVqTf-CnEC-enLmBxisRuCVsSwkbZddIbSc1fWHlGRp7MU80KhB0skZby1yBEk6H5xf0AbA9ok1z_3s4hUfezsu0$>! I'm a bit baffled by how we get the error message, but it looks like the problem is something to do with a weird field. Error -204 is VCZ_ERR_FIELD_UNSUPPORTED_NUM_COLUMNS <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/blob/475d7c915369d172a9b93d2ce0a912711c17cf5f/lib/vcf_encoder.h*L47C9-L47C46__;Iw!!K-Hz7m0Vt54!hnoAgJ_PVqTf-CnEC-enLmBxisRuCVsSwkbZddIbSc1fWHlGRp7MU80KhB0skZby1yBEk6H5xf0AbA9ok1z_3s4h4oxiu8E$> which only gets thrown in one place: https://github.com/sgkit-dev/vcztools/blob/475d7c915369d172a9b93d2ce0a912711c17cf5f/lib/vcf_encoder.c#L493 <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/blob/475d7c915369d172a9b93d2ce0a912711c17cf5f/lib/vcf_encoder.c*L493__;Iw!!K-Hz7m0Vt54!hnoAgJ_PVqTf-CnEC-enLmBxisRuCVsSwkbZddIbSc1fWHlGRp7MU80KhB0skZby1yBEk6H5xf0AbA9ok1z_3s4hUn3IUG8$> Which says the number of columns in the field is 0 or less. Would you mind putting in a bit more information here so we can debug this and handling it better for other UKB users? The output of vcf2zarr inspect test.vcz would be a great start. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/issues/245*issuecomment-3654629813__;Iw!!K-Hz7m0Vt54!hnoAgJ_PVqTf-CnEC-enLmBxisRuCVsSwkbZddIbSc1fWHlGRp7MU80KhB0skZby1yBEk6H5xf0AbA9ok1z_3s4hsbZ07_U$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AKDWBPD5SE3S6QTZ247KOS34BZ5OBAVCNFSM6AAAAACPA3UY36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNJUGYZDSOBRGM__;!!K-Hz7m0Vt54!hnoAgJ_PVqTf-CnEC-enLmBxisRuCVsSwkbZddIbSc1fWHlGRp7MU80KhB0skZby1yBEk6H5xf0AbA9ok1z_3s4hy8-9hfQ$> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jeromekelleher]

Hmmm - looks like this edge case has also triggered a bug in the vcf2zarr inspect logic, unfortunately. I think it's probably something to do with the call_FT and/or call_LAA fields.

It would be helpful here to see the schema that bio2zarr generates, and the ICF inspect will probably work. Can you run this please?

vcf2zarr explode -Q "${in_file}" "${icf_file}"
echo "====mkschema====" 1>&2
vcf2zarr mkschema "${icf_file}"
echo "====inspect ICF====" 1>&2
vcf2zarr inspect "${icf_file}"

Thanks for your help in debugging, I'm sure it'll be a simple fix once we've diagnosed.

[browning-lab]

Here's the mkschema and inspect ICF output: ``` ====mkschema==== { "format_version": "0.6", "dimensions": { "variants": { "size": 213, "chunk_size": 213 }, "samples": { "size": 490541, "chunk_size": 10000 }, "alleles": { "size": 5, "chunk_size": 5 }, "alt_alleles": { "size": 4, "chunk_size": 4 }, "filters": { "size": 24, "chunk_size": 24 }, "ploidy": { "size": 2, "chunk_size": 2 }, "genotypes": { "size": 15, "chunk_size": 15 }, "FORMAT_LAA_dim": { "size": 0, "chunk_size": 1 } }, "fields": [ { "name": "variant_contig", "dtype": "i2", "dimensions": [ "variants" ], "description": "An identifier from the reference genome or an angle-bracketed ID string pointing to a contig in the assembly file", "compressor": null, "filters": null, "source": null }, { "name": "variant_filter", "dtype": "bool", "dimensions": [ "variants", "filters" ], "description": "Filter status of the variant", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null }, { "name": "variant_allele", "dtype": "O", "dimensions": [ "variants", "alleles" ], "description": "List of the reference and alternate alleles", "compressor": null, "filters": null, "source": null }, { "name": "variant_length", "dtype": "i1", "dimensions": [ "variants" ], "description": "The length of the variant measured in bases", "compressor": null, "filters": null, "source": null }, { "name": "variant_id", "dtype": "O", "dimensions": [ "variants" ], "description": "List of unique identifiers where applicable", "compressor": null, "filters": null, "source": null }, { "name": "variant_id_mask", "dtype": "bool", "dimensions": [ "variants" ], "description": "", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null }, { "name": "variant_quality", "dtype": "f4", "dimensions": [ "variants" ], "description": "Phred-scaled quality score", "compressor": null, "filters": null, "source": "QUAL" }, { "name": "variant_position", "dtype": "i4", "dimensions": [ "variants" ], "description": "The reference position", "compressor": null, "filters": null, "source": "POS" }, { "name": "variant_AC", "dtype": "i4", "dimensions": [ "variants", "alt_alleles" ], "description": "Allele count in genotypes", "compressor": null, "filters": null, "source": "INFO/AC" }, { "name": "variant_AN", "dtype": "i4", "dimensions": [ "variants" ], "description": "Total number of alleles in called genotypes", "compressor": null, "filters": null, "source": "INFO/AN" }, { "name": "variant_NS", "dtype": "i4", "dimensions": [ "variants" ], "description": "Total number of samples", "compressor": null, "filters": null, "source": "INFO/NS" }, { "name": "variant_NS_GT", "dtype": "i4", "dimensions": [ "variants" ], "description": "Total number of samples with called genotypes", "compressor": null, "filters": null, "source": "INFO/NS_GT" }, { "name": "variant_NS_NOGT", "dtype": "i1", "dimensions": [ "variants" ], "description": "Total number of samples with unknown genotypes ./.", "compressor": null, "filters": null, "source": "INFO/NS_NOGT" }, { "name": "variant_NS_NODATA", "dtype": "i2", "dimensions": [ "variants" ], "description": "Total number of samples with no coverage", "compressor": null, "filters": null, "source": "INFO/NS_NODATA" }, { "name": "variant_IC", "dtype": "f4", "dimensions": [ "variants" ], "description": "The inbreeding coefficient", "compressor": null, "filters": null, "source": "INFO/IC" }, { "name": "variant_HWE", "dtype": "f4", "dimensions": [ "variants", "alt_alleles" ], "description": "The exact conditional Hardy-Weinberg Equilibrium P-value for each ALT allele", "compressor": null, "filters": null, "source": "INFO/HWE" }, { "name": "variant_ExcHet", "dtype": "f4", "dimensions": [ "variants", "alt_alleles" ], "description": "The exact conditional Excess Heterozygosity P-value for each ALT allele", "compressor": null, "filters": null, "source": "INFO/ExcHet" }, { "name": "variant_HWE_CHISQ", "dtype": "f4", "dimensions": [ "variants" ], "description": "The site-wise chi-squared Hardy-Weinberg Equilibrium P-value", "compressor": null, "filters": null, "source": "INFO/HWE_CHISQ" }, { "name": "variant_AF", "dtype": "f4", "dimensions": [ "variants", "alt_alleles" ], "description": "Allele frequency for each ALT allele", "compressor": null, "filters": null, "source": "INFO/AF" }, { "name": "call_GQ", "dtype": "i1", "dimensions": [ "variants", "samples" ], "description": "Genotype Quality", "compressor": null, "filters": null, "source": "FORMAT/GQ" }, { "name": "call_LAD", "dtype": "i1", "dimensions": [ "variants", "samples" ], "description": "Localized field: Allelic Depths", "compressor": null, "filters": null, "source": "FORMAT/LAD" }, { "name": "call_FT", "dtype": "O", "dimensions": [ "variants", "samples" ], "description": "Sample filter, 'PASS' indicates that all single sample filters passed for this sample", "compressor": null, "filters": null, "source": "FORMAT/FT" }, { "name": "call_LPL", "dtype": "i1", "dimensions": [ "variants", "samples" ], "description": "Local normalized, Phred-scaled likelihoods for genotypes as in original gVCF (without allele reordering)", "compressor": null, "filters": null, "source": "FORMAT/LPL" }, { "name": "call_LAA", "dtype": "O", "dimensions": [ "variants", "samples", "FORMAT_LAA_dim" ], "description": "Mapping of alt allele index from original gVCF to msVCF, comma-separated, 1-based (each value is the allele index in the msVCF)", "compressor": null, "filters": null, "source": "FORMAT/LAA" }, { "name": "call_LAF", "dtype": "f4", "dimensions": [ "variants", "samples", "alt_alleles" ], "description": "Local allele fractions for alt alleles in the order listed", "compressor": null, "filters": null, "source": "FORMAT/LAF" }, { "name": "call_QL", "dtype": "f4", "dimensions": [ "variants", "samples" ], "description": "Phred-scaled probability that the site has no variant in this sample (original gVCF QUAL)", "compressor": null, "filters": null, "source": "FORMAT/QL" }, { "name": "call_genotype_phased", "dtype": "bool", "dimensions": [ "variants", "samples" ], "description": "", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null }, { "name": "call_genotype", "dtype": "i1", "dimensions": [ "variants", "samples", "ploidy" ], "description": "", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null }, { "name": "call_genotype_mask", "dtype": "bool", "dimensions": [ "variants", "samples", "ploidy" ], "description": "", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null } ], "defaults": { "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 1, "blocksize": 0 }, "filters": [] } ``` ``` ====inspect ICF==== }name type chunks size compressed max_n min_val max_val -------------- ------- -------- --------- ------------ ------- --------- --------- CHROM String 2 46.59 KiB 398 bytes 1 n/a n/a POS Integer 2 46.59 KiB 974 bytes 1 1e+07 1e+07 QUAL Float 2 46.59 KiB 1.3 KiB 1 89 1e+02 ID String 2 46.59 KiB 3.43 KiB 1 n/a n/a FILTERS String 2 53.25 KiB 392 bytes 1 n/a n/a REF String 2 46.59 KiB 1 KiB 1 n/a n/a ALT String 2 49.38 KiB 1.37 KiB 4 n/a n/a rlen Integer 2 46.59 KiB 482 bytes 1 1 6 INFO/AC Integer 2 48.12 KiB 1.34 KiB 4 1 3.1e+05 INFO/AN Integer 2 46.59 KiB 676 bytes 1 9.8e+05 9.8e+05 INFO/NS Integer 2 46.59 KiB 380 bytes 1 4.9e+05 4.9e+05 INFO/NS_GT Integer 2 46.59 KiB 674 bytes 1 4.9e+05 4.9e+05 INFO/NS_NOGT Integer 2 46.59 KiB 376 bytes 1 0 0 INFO/NS_NODATA Integer 2 46.59 KiB 636 bytes 1 0 9.7e+02 INFO/IC Float 2 6.66 KiB 104 bytes 0 n/a n/a INFO/HWE Float 2 6.66 KiB 104 bytes 0 n/a n/a INFO/ExcHet Float 2 6.66 KiB 104 bytes 0 n/a n/a INFO/HWE_CHISQ Float 2 6.66 KiB 104 bytes 0 n/a n/a FORMAT/GT Integer 20 1.17 GiB 737.92 KiB 3 0 4 FORMAT/GQ Integer 2 6.66 KiB 104 bytes 0 n/a n/a FORMAT/LAD Integer 2 6.66 KiB 104 bytes 0 n/a n/a FORMAT/FT String 2 6.66 KiB 104 bytes 0 n/a n/a FORMAT/LPL Integer 2 6.66 KiB 104 bytes 0 n/a n/a FORMAT/LAA String 2 6.66 KiB 104 bytes 0 n/a n/a FORMAT/LAF Float 2 6.66 KiB 104 bytes 0 n/a n/a FORMAT/QL Float 2 6.66 KiB 104 bytes 0 n/a n/a INFO/AF Float 2 48.12 KiB 1.93 KiB 4 1e-06 0.31 ```

[jeromekelleher]

Reformatting the output here so it's a bit more legible:

name            type       chunks  size       compressed      max_n  min_val    max_val
--------------  -------  --------  ---------  ------------  ------- ---------  ---------
CHROM           String          2  46.59 KiB  398 bytes           1  n/a    n/a
POS             Integer         2  46.59 KiB  974 bytes           1  1e+07  1e+07
QUAL            Float           2  46.59 KiB  1.3 KiB             1  89     1e+02
ID              String          2  46.59 KiB  3.43 KiB            1  n/a    n/a
FILTERS         String          2  53.25 KiB  392 bytes           1  n/a    n/a
REF             String          2  46.59 KiB  1 KiB               1  n/a    n/a
ALT             String          2  49.38 KiB  1.37 KiB            4  n/a    n/a
rlen            Integer         2  46.59 KiB  482 bytes           1  1      6
INFO/AC         Integer         2  48.12 KiB  1.34 KiB            4  1      3.1e+05
INFO/AN         Integer         2  46.59 KiB  676 bytes           1  9.8e+05    9.8e+05
INFO/NS         Integer         2  46.59 KiB  380 bytes           1  4.9e+05    4.9e+05
INFO/NS_GT      Integer         2  46.59 KiB  674 bytes           1  4.9e+05    4.9e+05
INFO/NS_NOGT    Integer         2  46.59 KiB  376 bytes           1  0          0
INFO/NS_NODATA  Integer         2  46.59 KiB  636 bytes           1  0      9.7e+02
INFO/IC         Float           2  6.66 KiB   104 bytes           0  n/a    n/a
INFO/HWE        Float           2  6.66 KiB   104 bytes           0  n/a    n/a
INFO/ExcHet     Float           2  6.66 KiB   104 bytes           0  n/a    n/a
INFO/HWE_CHISQ  Float           2  6.66 KiB   104 bytes           0  n/a    n/a
FORMAT/GT       Integer        20  1.17 GiB   737.92 KiB          3  0      4
FORMAT/GQ       Integer         2  6.66 KiB   104 bytes           0  n/a    n/a
FORMAT/LAD      Integer         2  6.66 KiB   104 bytes           0  n/a    n/a 
FORMAT/FT       String          2  6.66 KiB   104 bytes           0  n/a    n/a
FORMAT/LPL      Integer         2  6.66 KiB   104 bytes           0  n/a    n/a
FORMAT/LAA      String          2  6.66 KiB   104 bytes           0  n/a    n/a
FORMAT/LAF      Float           2  6.66 KiB   104 bytes           0  n/a    n/a
FORMAT/QL       Float           2  6.66 KiB   104 bytes           0  n/a    n/a
INFO/AF         Float           2  48.12 KiB  1.93 KiB            4  1e-06  0.31

[jeromekelleher]

The problem here is almost certainly due to a bunch of fields being declared in the header with an unknown inner dimension which have no data in the body (as shown by the n/a values for max and min, and a max_n of 0). To see if we can at least get the fields that have data to work, I've trimmed down the schema to remove them @browning-lab:

{
    "format_version": "0.6",
    "dimensions": {
        "variants": {
            "size": 213,
            "chunk_size": 213
        },
        "samples": {
            "size": 490541,
            "chunk_size": 10000
        },
        "alleles": {
            "size": 5,
            "chunk_size": 5
        },
        "alt_alleles": {
            "size": 4,
            "chunk_size": 4
        },
        "filters": {
            "size": 24,
            "chunk_size": 24
        },
        "ploidy": {
            "size": 2,
            "chunk_size": 2
        },
        "genotypes": {
            "size": 15,
            "chunk_size": 15
        },
        "FORMAT_LAA_dim": {
            "size": 0,
            "chunk_size": 1
        }
    },
    "fields": [
        {
            "name": "variant_contig",
            "dtype": "i2",
            "dimensions": [
                "variants"
            ],
            "description": "An identifier from the reference genome or an angle-bracketed ID string pointing to a contig in the assembly file",
            "compressor": null,
            "filters": null,
            "source": null
        },
        {
            "name": "variant_filter",
            "dtype": "bool",
            "dimensions": [
                "variants",
                "filters"
            ],
            "description": "Filter status of the variant",
            "compressor": {
                "id": "blosc",
                "cname": "zstd",
                "clevel": 7,
                "shuffle": 2,
                "blocksize": 0
            },
            "filters": null,
            "source": null
        },
        {
            "name": "variant_allele",
            "dtype": "O",
            "dimensions": [
                "variants",
                "alleles"
            ],
            "description": "List of the reference and alternate alleles",
            "compressor": null,
            "filters": null,
            "source": null
        },
        {
            "name": "variant_length",
            "dtype": "i1",
            "dimensions": [
                "variants"
            ],
            "description": "The length of the variant measured in bases",
            "compressor": null,
            "filters": null,
            "source": null
        },
        {
            "name": "variant_id",
            "dtype": "O",
            "dimensions": [
                "variants"
            ],
            "description": "List of unique identifiers where applicable",
            "compressor": null,
            "filters": null,
            "source": null
        },
        {
            "name": "variant_id_mask",
            "dtype": "bool",
            "dimensions": [
                "variants"
            ],
            "description": "",
            "compressor": {
                "id": "blosc",
                "cname": "zstd",
                "clevel": 7,
                "shuffle": 2,
                "blocksize": 0
            },
            "filters": null,
            "source": null
        },
        {
            "name": "variant_quality",
            "dtype": "f4",
            "dimensions": [
                "variants"
            ],
            "description": "Phred-scaled quality score",
            "compressor": null,
            "filters": null,
            "source": "QUAL"
        },
        {
            "name": "variant_position",
            "dtype": "i4",
            "dimensions": [
                "variants"
            ],
            "description": "The reference position",
            "compressor": null,
            "filters": null,
            "source": "POS"
        },
        {
            "name": "variant_AC",
            "dtype": "i4",
            "dimensions": [
                "variants",
                "alt_alleles"
            ],
            "description": "Allele count in genotypes",
            "compressor": null,
            "filters": null,
            "source": "INFO/AC"
        },
        {
            "name": "variant_AN",
            "dtype": "i4",
            "dimensions": [
                "variants"
            ],
            "description": "Total number of alleles in called genotypes",
            "compressor": null,
            "filters": null,
            "source": "INFO/AN"
        },
        {
            "name": "variant_NS",
            "dtype": "i4",
            "dimensions": [
                "variants"
            ],
            "description": "Total number of samples",
            "compressor": null,
            "filters": null,
            "source": "INFO/NS"
        },
        {
            "name": "variant_NS_GT",
            "dtype": "i4",
            "dimensions": [
                "variants"
            ],
            "description": "Total number of samples with called genotypes",
            "compressor": null,
            "filters": null,
            "source": "INFO/NS_GT"
        },
        {
            "name": "variant_NS_NOGT",
            "dtype": "i1",
            "dimensions": [
                "variants"
            ],
            "description": "Total number of samples with unknown genotypes ./.",
            "compressor": null,
            "filters": null,
            "source": "INFO/NS_NOGT"
        },
        {
            "name": "variant_NS_NODATA",
            "dtype": "i2",
            "dimensions": [
                "variants"
            ],
            "description": "Total number of samples with no coverage",
            "compressor": null,
            "filters": null,
            "source": "INFO/NS_NODATA"
        },
        {
            "name": "variant_AF",
            "dtype": "f4",
            "dimensions": [
                "variants",
                "alt_alleles"
            ],
            "description": "Allele frequency for each ALT allele",
            "compressor": null,
            "filters": null,
            "source": "INFO/AF"
        },
        {
            "name": "call_genotype_phased",
            "dtype": "bool",
            "dimensions": [
                "variants",
                "samples"
            ],
            "description": "",
            "compressor": {
                "id": "blosc",
                "cname": "zstd",
                "clevel": 7,
                "shuffle": 2,
                "blocksize": 0
            },
            "filters": null,
            "source": null
        },
        {
            "name": "call_genotype",
            "dtype": "i1",
            "dimensions": [
                "variants",
                "samples",
                "ploidy"
            ],
            "description": "",
            "compressor": {
                "id": "blosc",
                "cname": "zstd",
                "clevel": 7,
                "shuffle": 2,
                "blocksize": 0
            },
            "filters": null,
            "source": null
        },
        {
            "name": "call_genotype_mask",
            "dtype": "bool",
            "dimensions": [
                "variants",
                "samples",
                "ploidy"
            ],
            "description": "",
            "compressor": {
                "id": "blosc",
                "cname": "zstd",
                "clevel": 7,
                "shuffle": 2,
                "blocksize": 0
            },
            "filters": null,
            "source": null
        }
    ],
    "defaults": {
        "compressor": {
            "id": "blosc",
            "cname": "zstd",
            "clevel": 7,
            "shuffle": 1,
            "blocksize": 0
        },
        "filters": []
    }
}

Can you try saving this as schema.json please @browning-lab, and then running the encode step using -s schema.json? This should convert the actual data.

I'll see if I can reproduce the problem and open an issue to track.

It's worth noting here that you're likely not going to get very impressive results with only 213 variants - I think you'd need to convert at least 10 of these segments together to get anything representative.

[browning-lab]

The conversion of zarr format to vcf.gz runs successfully with the edited schema. Thank you for finding this workaround and for the quick resolution. The purpose of the 1 kb input region is to validate my pipeline. I plan to use at a 10 Mb or longer region to measure the vcz compression ratio and compression/decompression compute times.

…

On Wed, Dec 17, 2025 at 3:04 AM Jerome Kelleher ***@***.***> wrote: *jeromekelleher* left a comment (sgkit-dev/vcztools#245) <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/issues/245*issuecomment-3664836850__;Iw!!K-Hz7m0Vt54!i2C62ART2FdN2tJtWfZb-NptV5MAxQJetE82vkfLVvitUz0rAFt8h4smAixPHrDIBWpXt6cFPQzPfgz2NPOHBcGzCfbOfSU$> The problem here is almost certainly due to a bunch of fields being declared in the header with an unknown inner dimension which have no data in the body (as shown by the n/a values for max and min, and a max_n of 0). To see if we can at least get the fields that have data to work, I've trimmed down the schema to remove them @browning-lab <https://urldefense.com/v3/__https://github.com/browning-lab__;!!K-Hz7m0Vt54!i2C62ART2FdN2tJtWfZb-NptV5MAxQJetE82vkfLVvitUz0rAFt8h4smAixPHrDIBWpXt6cFPQzPfgz2NPOHBcGzIXFEab8$> : { "format_version": "0.6", "dimensions": { "variants": { "size": 213, "chunk_size": 213 }, "samples": { "size": 490541, "chunk_size": 10000 }, "alleles": { "size": 5, "chunk_size": 5 }, "alt_alleles": { "size": 4, "chunk_size": 4 }, "filters": { "size": 24, "chunk_size": 24 }, "ploidy": { "size": 2, "chunk_size": 2 }, "genotypes": { "size": 15, "chunk_size": 15 }, "FORMAT_LAA_dim": { "size": 0, "chunk_size": 1 } }, "fields": [ { "name": "variant_contig", "dtype": "i2", "dimensions": [ "variants" ], "description": "An identifier from the reference genome or an angle-bracketed ID string pointing to a contig in the assembly file", "compressor": null, "filters": null, "source": null }, { "name": "variant_filter", "dtype": "bool", "dimensions": [ "variants", "filters" ], "description": "Filter status of the variant", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null }, { "name": "variant_allele", "dtype": "O", "dimensions": [ "variants", "alleles" ], "description": "List of the reference and alternate alleles", "compressor": null, "filters": null, "source": null }, { "name": "variant_length", "dtype": "i1", "dimensions": [ "variants" ], "description": "The length of the variant measured in bases", "compressor": null, "filters": null, "source": null }, { "name": "variant_id", "dtype": "O", "dimensions": [ "variants" ], "description": "List of unique identifiers where applicable", "compressor": null, "filters": null, "source": null }, { "name": "variant_id_mask", "dtype": "bool", "dimensions": [ "variants" ], "description": "", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null }, { "name": "variant_quality", "dtype": "f4", "dimensions": [ "variants" ], "description": "Phred-scaled quality score", "compressor": null, "filters": null, "source": "QUAL" }, { "name": "variant_position", "dtype": "i4", "dimensions": [ "variants" ], "description": "The reference position", "compressor": null, "filters": null, "source": "POS" }, { "name": "variant_AC", "dtype": "i4", "dimensions": [ "variants", "alt_alleles" ], "description": "Allele count in genotypes", "compressor": null, "filters": null, "source": "INFO/AC" }, { "name": "variant_AN", "dtype": "i4", "dimensions": [ "variants" ], "description": "Total number of alleles in called genotypes", "compressor": null, "filters": null, "source": "INFO/AN" }, { "name": "variant_NS", "dtype": "i4", "dimensions": [ "variants" ], "description": "Total number of samples", "compressor": null, "filters": null, "source": "INFO/NS" }, { "name": "variant_NS_GT", "dtype": "i4", "dimensions": [ "variants" ], "description": "Total number of samples with called genotypes", "compressor": null, "filters": null, "source": "INFO/NS_GT" }, { "name": "variant_NS_NOGT", "dtype": "i1", "dimensions": [ "variants" ], "description": "Total number of samples with unknown genotypes ./.", "compressor": null, "filters": null, "source": "INFO/NS_NOGT" }, { "name": "variant_NS_NODATA", "dtype": "i2", "dimensions": [ "variants" ], "description": "Total number of samples with no coverage", "compressor": null, "filters": null, "source": "INFO/NS_NODATA" }, { "name": "variant_AF", "dtype": "f4", "dimensions": [ "variants", "alt_alleles" ], "description": "Allele frequency for each ALT allele", "compressor": null, "filters": null, "source": "INFO/AF" }, { "name": "call_genotype_phased", "dtype": "bool", "dimensions": [ "variants", "samples" ], "description": "", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null }, { "name": "call_genotype", "dtype": "i1", "dimensions": [ "variants", "samples", "ploidy" ], "description": "", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null }, { "name": "call_genotype_mask", "dtype": "bool", "dimensions": [ "variants", "samples", "ploidy" ], "description": "", "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 2, "blocksize": 0 }, "filters": null, "source": null } ], "defaults": { "compressor": { "id": "blosc", "cname": "zstd", "clevel": 7, "shuffle": 1, "blocksize": 0 }, "filters": [] } } Can you try saving this as schema.json please @browning-lab <https://urldefense.com/v3/__https://github.com/browning-lab__;!!K-Hz7m0Vt54!i2C62ART2FdN2tJtWfZb-NptV5MAxQJetE82vkfLVvitUz0rAFt8h4smAixPHrDIBWpXt6cFPQzPfgz2NPOHBcGzIXFEab8$>, and then running the encode step using -s schema.json? This should convert the actual data. I'll see if I can reproduce the problem and open an issue to track. It's worth noting here that you're likely not going to get very impressive results with only 213 variants - I think you'd need to convert at least 10 of these segments together to get anything representative. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/issues/245*issuecomment-3664836850__;Iw!!K-Hz7m0Vt54!i2C62ART2FdN2tJtWfZb-NptV5MAxQJetE82vkfLVvitUz0rAFt8h4smAixPHrDIBWpXt6cFPQzPfgz2NPOHBcGzCfbOfSU$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AKDWBPHOQJZFZOMRMTCU5EL4CE2DNAVCNFSM6AAAAACPA3UY36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNRUHAZTMOBVGA__;!!K-Hz7m0Vt54!i2C62ART2FdN2tJtWfZb-NptV5MAxQJetE82vkfLVvitUz0rAFt8h4smAixPHrDIBWpXt6cFPQzPfgz2NPOHBcGzi8k-xPE$> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jeromekelleher]

Great, glad to hear it @browning-lab. Happy to help if you hit any other stumbling blocks.

We've not tried this out on the RAP at all, so I'm not sure how well things will perform - there may well need to be some tweaking to get things working well in this environment.

[browning-lab]

I next attempted to compress a 20 Mb interval (chr20:1-20000000) of phased UKB whole genome sequence data and got an error. The job was run on a 8-core VM (mem3_ssd1_v2_x8), and I used the commands: time ( vcf2zarr explode -Q -p 8 "${in_file}" "${icf_file}" vcf2zarr encode -Q -p 8 -s schema.json "${icf_file}" "${zarr_dir}" ) where "schema.json" is the cleaned-up schema that you sent in your earlier email. These commands ran successfully on my 1kb interval test data. Thank you for your help so far. I need to switch over and resume working on some other projects now, but FYI, here is the error message that was generated. concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.12/concurrent/futures/process.py", line 263, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/bio2zarr/vcz.py", line 778, in encode_partition self.encode_alleles_and_genotypes_partition(partition_index) File "/usr/local/lib/python3.12/dist-packages/bio2zarr/vcz.py", line 846, in encode_alleles_and_genotypes_partition for variant_data in self.source.iter_alleles_and_genotypes( File "/usr/local/lib/python3.12/dist-packages/bio2zarr/vcf.py", line 1073, in iter_alleles_and_genotypes for variant_length, alleles, (gt, phased) in zip( File "/usr/local/lib/python3.12/dist-packages/bio2zarr/vcf.py", line 1049, in iter_alleles alleles[1 : 1 + len(alt)] = alt ~~~~~~~^^^^^^^^^^^^^^^^^^ ValueError: could not broadcast input array from shape (5,) into shape (4,) """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/bin/vcf2zarr", line 8, in <module> sys.exit(vcf2zarr_main()) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1485, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1406, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1873, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1269, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 824, in invoke return callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/bio2zarr/cli.py", line 393, in encode vcf_mod.encode( File "/usr/local/lib/python3.12/dist-packages/bio2zarr/vcf.py", line 1682, in encode vzw.encode_all_partitions( File "/usr/local/lib/python3.12/dist-packages/bio2zarr/vcz.py", line 1104, in encode_all_partitions with core.ParallelWorkManager(num_workers, progress_config) as pwm: File "/usr/local/lib/python3.12/dist-packages/bio2zarr/core.py", line 352, in __exit__ wait_on_futures(self.futures) File "/usr/local/lib/python3.12/dist-packages/bio2zarr/core.py", line 163, in wait_on_futures raise exception ValueError: could not broadcast input array from shape (5,) into shape (4,) real 864m8.509s user 6344m15.241s sys 253m16.772s

…

On Thu, Dec 18, 2025 at 6:14 AM Jerome Kelleher ***@***.***> wrote: *jeromekelleher* left a comment (sgkit-dev/vcztools#245) <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/issues/245*issuecomment-3670491688__;Iw!!K-Hz7m0Vt54!iEjb1BpLHsadEvCmQ68r3dvqTC3KwxvSkw76NPk-IqRl8DWeOZwOPR9vEcgl90Bcto_8VatpVYKcj0fAhwwh-Lq5xLlYu54$> Great, glad to hear it @browning-lab <https://urldefense.com/v3/__https://github.com/browning-lab__;!!K-Hz7m0Vt54!iEjb1BpLHsadEvCmQ68r3dvqTC3KwxvSkw76NPk-IqRl8DWeOZwOPR9vEcgl90Bcto_8VatpVYKcj0fAhwwh-Lq5E8Kb9TE$>. Happy to help if you hit any other stumbling blocks. We've not tried this out on the RAP at all, so I'm not sure how well things will perform - there may well need to be some tweaking to get things working well in this environment. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/issues/245*issuecomment-3670491688__;Iw!!K-Hz7m0Vt54!iEjb1BpLHsadEvCmQ68r3dvqTC3KwxvSkw76NPk-IqRl8DWeOZwOPR9vEcgl90Bcto_8VatpVYKcj0fAhwwh-Lq5xLlYu54$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AKDWBPG2PFLBCXBAWYGCJKL4CKZEXAVCNFSM6AAAAACPA3UY36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNZQGQ4TCNRYHA__;!!K-Hz7m0Vt54!iEjb1BpLHsadEvCmQ68r3dvqTC3KwxvSkw76NPk-IqRl8DWeOZwOPR9vEcgl90Bcto_8VatpVYKcj0fAhwwh-Lq51_i3eFQ$> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jeromekelleher]

The problem here is that the schema includes the dataset shape, which was based on your smaller subset @browning-lab .

To get this work work in general, it's probably simplest to filter out the fields you don't want using jq as demonstrated in the documentation here.

It is generally easier if you do things in two passes though - so run vcf2zarr explode as one job first, and then run the encode afterwards. That way you can tweak the schema manually in between without having to do so programatically.

Sorry, this is much more awkward than it should be - the schema tools are quite primitive currently.

[browning-lab]

I see. Thank you for this diagnosis and workaround.

…

On Fri, Dec 19, 2025 at 5:15 AM Jerome Kelleher ***@***.***> wrote: *jeromekelleher* left a comment (sgkit-dev/vcztools#245) <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/issues/245*issuecomment-3675050622__;Iw!!K-Hz7m0Vt54!gz5ue8oHl_OH8ESQqxY_2ZOgqZcivYKd37Ub1dMVoeYTwmfrIUSpp7nk3ZcD3FBDTNH8f2-DjZMjduwGTuBlIBnGHO7trrU$> The problem here is that the schema includes the dataset shape, which was based on your smaller subset @browning-lab <https://urldefense.com/v3/__https://github.com/browning-lab__;!!K-Hz7m0Vt54!gz5ue8oHl_OH8ESQqxY_2ZOgqZcivYKd37Ub1dMVoeYTwmfrIUSpp7nk3ZcD3FBDTNH8f2-DjZMjduwGTuBlIBnGzg-Hnmw$> . To get this work work in general, it's probably simplest to filter out the fields you don't want using jq as demonstrated in the documentation here <https://urldefense.com/v3/__https://sgkit-dev.github.io/bio2zarr/vcf2zarr/tutorial.html*medium-dataset__;Iw!!K-Hz7m0Vt54!gz5ue8oHl_OH8ESQqxY_2ZOgqZcivYKd37Ub1dMVoeYTwmfrIUSpp7nk3ZcD3FBDTNH8f2-DjZMjduwGTuBlIBnGxoLIl4I$> . It is generally easier if you do things in two passes though - so run vcf2zarr explode as one job first, and then run the encode afterwards. That way you can tweak the schema manually in between without having to do so programatically. Sorry, this is much more awkward than it should be - the schema tools are quite primitive currently. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/sgkit-dev/vcztools/issues/245*issuecomment-3675050622__;Iw!!K-Hz7m0Vt54!gz5ue8oHl_OH8ESQqxY_2ZOgqZcivYKd37Ub1dMVoeYTwmfrIUSpp7nk3ZcD3FBDTNH8f2-DjZMjduwGTuBlIBnGHO7trrU$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AKDWBPFE7DATT3J4ZZFUA7T4CP267AVCNFSM6AAAAACPA3UY36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNZVGA2TANRSGI__;!!K-Hz7m0Vt54!gz5ue8oHl_OH8ESQqxY_2ZOgqZcivYKd37Ub1dMVoeYTwmfrIUSpp7nk3ZcD3FBDTNH8f2-DjZMjduwGTuBlIBnGSfbg32Q$> . You are receiving this because you were mentioned.Message ID: ***@***.***>