Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Inaccessible externally due to container ip registration when deploying with Docker #318

Open
1 of 2 tasks
haishui126 opened this issue Jan 9, 2025 · 5 comments · May be fixed by #319
Open
1 of 2 tasks

[Bug] Inaccessible externally due to container ip registration when deploying with Docker #318

haishui126 opened this issue Jan 9, 2025 · 5 comments · May be fixed by #319
Labels
bug Something isn't working

Comments

@haishui126
Copy link

haishui126 commented Jan 9, 2025

Search before asking

  • I searched in the issues and found nothing similar.

Fluss version

0.5.0

Minimal reproduce step

  1. get docker compose yml: https://alibaba.github.io/fluss-docs/docs/install-deploy/deploying-with-docker/#create-docker-composeyml-file
  2. add ports: - "9123:9123" for coordinator-server and deploy to server
  3. create fluss database at local

What doesn't meet your expectations?

coordinator-server log:
2025-01-09 03:12:19,633 INFO com.alibaba.fluss.rpc.netty.server.NettyServer [] - Successfully start Netty server (took 96 ms). Listening on SocketAddress /192.168.0.3:9123.
2025-01-09 03:12:19,734 INFO com.alibaba.fluss.server.zk.ZooKeeperClient [] - Registered leader CoordinatorAddress{id='0d145c86-9a97-41c2-9341-79b6db9206e4', host='192.168.0.3', port=9123} at path /coordinators/active.

sql-client error log:
Caused by: com.alibaba.fluss.exception.NetworkException: Disconnected from node 192.168.0.3:9123 (id: cs--1)

192.168.0.3 is the container IP, not the server IP.

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@haishui126 haishui126 added the bug Something isn't working label Jan 9, 2025
@haishui126 haishui126 changed the title [Bug] Fluss inaccessible externally due to container ip registration when deploying with Docker [Bug] Inaccessible externally due to container ip registration when deploying with Docker Jan 9, 2025
@michaelkoepf
Copy link
Contributor

michaelkoepf commented Jan 9, 2025

Hi @haishui126,

  1. do you use the docker compose manifest for one or multiple tablet servers? if it is the one for one tablet server, there seems to be an error in the docker compose manifest. the command of the tablet-server should be command: tabletServer and not command: tabletServer:0.5.0.

Fixed docker-compose.yml for a single tablet server:

services:
  coordinator-server:
    image: fluss/fluss:0.5.0
    command: coordinatorServer
    depends_on:
      - zookeeper
    environment:
      - |
        FLUSS_PROPERTIES=
        zookeeper.address: zookeeper:2181
        coordinator.host: coordinator-server
        remote.data.dir: /tmp/fluss/remote-data
  tablet-server:
    image: fluss/fluss
    command: tabletServer  # << LINE TO BE FIXED
    depends_on:
      - coordinator-server
    environment:
      - |
        FLUSS_PROPERTIES=
        zookeeper.address: zookeeper:2181
        tablet-server.host: tablet-server
        tablet-server.id: 0
        kv.snapshot.interval: 0s
        data.dir: /tmp/fluss/data
        remote.data.dir: /tmp/fluss/remote-data
    volumes:
      - shared-tmpfs:/tmp/fluss
  zookeeper:
    restart: always
    image: zookeeper:3.9.2

volumes:
  shared-tmpfs:
    driver: local
    driver_opts:
      type: "tmpfs"
      device: "tmpfs"

Run docker compose up -d and check with docker ps -a if all servers are running.


  1. Regarding this

add ports: - "9123:9123" for coordinator-server and deploy to server

and this

sql-client error log: Caused by: com.alibaba.fluss.exception.NetworkException: Disconnected from node 192.168.0.3:9123 (id: cs--1)

can you explain in more detail what you are trying to achieve? does this guide solve your problem?

@haishui126
Copy link
Author

Hi @michaelkoepf.
I fixed the command before I started the service. My compose file is the same with you.
I added ports: - "9123:9123" to enable access to the Fluss service from outside my server.

My server's external IP address is 10.xx.xx.xx, but the host address registered to zk is 192.168.xx.xx, which is the container ip.
The ip registered to zk is from com.alibaba.fluss.rpc.netty.server.NettyServer#getHostname.

This guide works well, beceuse flink and fluss is in the same docker service. They can communicate with each other using their container IP.

@michaelkoepf
Copy link
Contributor

michaelkoepf commented Jan 9, 2025

ok. this means you run fluss with the docker compose manifest and flink and the sql client somewhere outside (on the same or even a different host), right?

how do you create the fluss catalog in the sql client? have you adapted the bootstrap.server option accordingly (see below)?

CREATE CATALOG my_fluss WITH (
    'type' = 'fluss',
    'bootstrap.servers' = '10.XXX.XXX.XXX:9123'
);

@haishui126
Copy link
Author

ok. this means you run fluss with the docker compose manifest and flink and the sql client somewhere outside (on the same or even a different host), right?

how do you create the fluss catalog in the sql client? have you adapted the bootstrap.server option accordingly (see below)?

CREATE CATALOG my_fluss WITH (
    'type' = 'fluss',
    'bootstrap.servers' = '10.XXX.XXX.XXX:9123'
);

right. the following is my code, the connection timed out when creating the database

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
        tableEnv.executeSql("CREATE CATALOG fluss_catalog WITH (\n" +
                "   'type' = 'fluss',\n" +
                "   'bootstrap.servers' = '10.xxx.xxx.xxx:9123'\n" +
                ");\n");

        tableEnv.executeSql("USE CATALOG `fluss_catalog`;");
        tableEnv.executeSql("CREATE DATABASE `db_test`;");

these are packets I captured with Wireshark, and you can see that the IP of CS and TS is obtained in 2491 and then tries to establish a new connection with CS 192.168.0.3.
image

I debugged the process of creating the database, and found that the IP obtained in MetadataUpdater is not 10.xxx.xxx.xxx, and finally I can locate the hostname obtained from InetSocketAddress in NettyServer cannot be accessed externally. I've also tried modifying 'coordinator.host', setting it to 10.xxx.xxx.xxx will cause the service to fail to start.

I also tried Deploying Local Cluster. Whether coordinator.host is set to localhost or 10.xxx.xxx.xxx, the IP registered to ZK is the private IP of my server, which my PC can't access.
I have not found a way to access Fluss outside of the same local area network.

@luoyuxia
Copy link
Collaborator

@haishui126 Thanks for reporting... Yes, you can't access Fluss from an extern network.. Fluss will need to support advertised.listeners like kafka.. See more detail in this article

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants