[Bug]: (External Catalog) Iceberg rest catalog implementation error

[Shreyas220]

What happened?

I am trying to connect to the external iceberg catalog
using this setup here along with amoro https://iceberg.apache.org/spark-quickstart/#docker-compose

using this jar https://github.com/databricks/iceberg-rest-image for catalog implementation

but getting this error

java.lang.IllegalArgumentException: Cannot initialize Catalog implementation 
/usr/local/amoro/lib/iceberg-rest-image-all.jar: Cannot find constructor for interface
org.apache.iceberg.catalog.Catalog Missing /usr/local/amoro/lib/iceberg-rest-image-all.jar
[java.lang.ClassNotFoundException: /usr/local/amoro/lib/iceberg-rest-image-all.jar]

Affects Versions

latest amoro imge

What table formats are you seeing the problem on?

Iceberg

What engines are you seeing the problem on?

AMS

How to reproduce

use this docker-compose https://iceberg.apache.org/spark-quickstart/#docker-compose
along with amoro

and try connecting to iceberg rest catalog

version: "3"

services:
  spark-iceberg:
    image: tabulario/spark-iceberg
    container_name: spark-iceberg
    build: spark/
    networks:
      iceberg_net:
    depends_on:
      - rest
      - minio
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    ports:
      - 8888:8888
      - 8080:8080
      - 10000:10000
      - 10001:10001
  rest:
    image: tabulario/iceberg-rest
    container_name: iceberg-rest
    networks:
      iceberg_net:
    ports:
      - 8181:8181
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
      - CATALOG_WAREHOUSE=s3://warehouse/
      - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
      - CATALOG_S3_ENDPOINT=http://minio:9000
  minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=admin
      - MINIO_ROOT_PASSWORD=password
      - MINIO_DOMAIN=minio
    networks:
      iceberg_net:
        aliases:
          - warehouse.minio
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"]
  mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    networks:
      iceberg_net:
    environment:
      - AWS_ACCESS_KEY_ID=admin
      - AWS_SECRET_ACCESS_KEY=password
      - AWS_REGION=us-east-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc rm -r --force minio/warehouse;
      /usr/bin/mc mb minio/warehouse;
      /usr/bin/mc policy set public minio/warehouse;
      tail -f /dev/null
      "
networks:
  iceberg_net:

Relevant log output

No response

Anything else

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Shreyas220]

if this is not an issue please guide what jar to use catalog impl for iceberg rest catalog

[klion26]

@Shreyas220, thanks for reporting the issue here. Could you please try to set the value of catalog-impl to the full class name of the rest catalog, and put the jar into $AMORO_HOME/lib dir?

[XBaith]

@Shreyas220 catalog-impl should be org.apache.iceberg.rest.RESTCatalog and you also missing some required properties like uri. Custom catalog type actually invoke CatalogUtil.loadCatalog, thus you need to fill in native icebreg catalog properties, refer to https://iceberg.apache.org/javadoc/1.6.0/org/apache/iceberg/CatalogProperties.html

[Shreyas220]

Hey @XBaith, @klion26 thanks alot 🙌 , got amoro with rest iceberg working

Now the next step is connecting it with Glue catalog, can also point me to what properties would be required for it ?

also is it possible to use s3 storage when using hive metastore ?