Note that revisions to the core of ianvs, including controllers, are usually about adding new algorithm schemes, e.g., creating a scheme for lifelong learning.

In this proposal, since single-task learning exists for large models, my suggestion is to consider adding examples as a priority, i.e., a new example of single-task learning, instead of changing the core of ianvs. That can also release the burden of implementation and review, by avoiding the impact on other examples, without ianvs core revision.

Recently, multi-modal data types are mostly supported. We might make better use of current data types, especailly under limited development time. Please refer to detailed comments on dataset handling below.

Benchmarks like metrics should be in examples instead of controllers. There are cases that metrics of the same name have different implementations in different scenarios, e.g., F1-score, BWT, etc.

"main.py" is forbidden in the core of ianvs and is not safe enough even as an example of ianvs.

As in the quick start of ianvs, we recommend launching ianvs using its command lines,
ianvs -f ./examples/pcb-aoi/singletask_learning_bench/benchmarkingjob.yaml

Structured datasets are constructed using .csv. For Unstructured Data,

1. datasets of image and audio are constructed using data index, i.e., URL with .txt.
2. datasets of natural language are constructed using .jsonl

In the current stage, it is not a good idea to add more data types that need to change codes in sedna before ianvs. My suggestion is to make better use of the current implementation.

For your reference,

1. Unstructured Data implementation using .txt:

• An image example is ready in Sedna federated learning
• Video-input examples are ready in Sedna incremental learning and joint inference.

2. Unstructured Data implementation using .jsonl:

• An NLP example from @IcyFeather233 is available in ianvs Single task learning for LLM with proposal and implementation.

When necessary, @aryan0931 might refer to @IcyFeather233 for more usage information on data types of ianvs LLM benchmarks. The implementation from @IcyFeather233 has already been successfully used in several members' projects merged in ianvs recently.

As mentioned above, metrics should be implemented in examples to avoid impacts on others examples.

The usage of metrics is also in ianvs examples with testenv.yaml. An example is available in ianvs documents, as the following.

# testenv.yaml
testenv:
...

# metric used for model evaluation
model_metric:
  # metric name; string type;
  name: "f1_score"
  # the url address of python file
  url: "./examples/pcb-aoi/incremental_learning_bench/testenv/f1_score.py"