magnetic one example failed with MultimodalWebSurfer is not defined

[davidqqq]

What happened?

While running example.py of magnetic one, "Click a button" step failed due to the following

Web surfing error:

Traceback (most recent call last):
  File \"/autogen/python/packages/autogen-magentic-one/src/autogen_magentic_one/agents/multimodal_web_surfer/multimodal_web_surfer.py\", line 285, in _generate_reply
    request_halt, content = await self.__generate_reply(cancellation_token)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/autogen/python/packages/autogen-magentic-one/src/autogen_magentic_one/agents/multimodal_web_surfer/multimodal_web_surfer.py\", line 594, in __generate_reply
    return await self._execute_tool(message, rects, tool_names, cancellation_token=cancellation_token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/autogen/python/packages/autogen-magentic-one/src/autogen_magentic_one/agents/multimodal_web_surfer/multimodal_web_surfer.py\", line 414, in _execute_tool
    page_metadata = json.dumps(await self._get_page_metadata(), indent=4)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/autogen/python/packages/autogen-magentic-one/src/autogen_magentic_one/agents/multimodal_web_surfer/multimodal_web_surfer.py\", line 643, in _get_page_metadata
    result = await self._page.evaluate(\"MultimodalWebSurfer.getPageMetadata();\")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/Users/davidtseng/miniconda3/envs/autogen/lib/python3.11/site-packages/playwright/async_api/_generated.py\", line 8491, in evaluate
    await self._impl_obj.evaluate(
  File \"/Users/davidtseng/miniconda3/envs/autogen/lib/python3.11/site-packages/playwright/_impl/_page.py\", line 469, in evaluate
    return await self._main_frame.evaluate(expression, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/Users/davidtseng/miniconda3/envs/autogen/lib/python3.11/site-packages/playwright/_impl/_frame.py\", line 278, in evaluate
    await self._channel.send(
  File \"/Users/davidtseng/miniconda3/envs/autogen/lib/python3.11/site-packages/playwright/_impl/_connection.py\", line 59, in send
    return await self._connection.wrap_api_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/Users/davidtseng/miniconda3/envs/autogen/lib/python3.11/site-packages/playwright/_impl/_connection.py\", line 520, in wrap_api_call
    raise rewrite_error(error, f\"{parsed_st['apiName']}: {error}\") from None\nplaywright._impl._errors.Error: Page.evaluate: ReferenceError: MultimodalWebSurfer is not defined
    at eval (eval at evaluate (:234:30), <anonymous>:1:1)
    at eval (<anonymous>)
    at UtilityScript.evaluate (<anonymous>:234:30)
    at UtilityScript.<anonymous> (<anonymous>:1:44)

What did you expect to happen?

No error

How can we reproduce it (as minimally and precisely as possible)?

Run exmple.py unmodified
User input: Visit https://smol.com and find out how much fabric conditioner cost.

AutoGen version

0.4

Which package was this bug in

Magentic One

Model used

gpt-4o

Python version

3.11

Operating system

MacOS

Any additional info you think would be helpful for fixing this bug

I suspect page transition after "clicking a button" action resulted in script calling MultimodalWebSurfer too early which is yet to initialize.

[mdazfar2]

@victordibia I will start working on this issue to resolve the error related to MultimodalWebSurfer. Stay tuned for updates as I investigate and implement a fix.

[davidqqq]

@mdazfar2 thanks for the fast PR! I failed to understand what changes in your PR fixed the issue. Can you explain a bit?

[mdazfar2]

@mdazfar2 thanks for the fast PR! I failed to understand what changes in your PR fixed the issue. Can you explain a bit?

@davidqqq It's my mistake. The PR I created doesn't make any sense. You can continue your work on it.

[davidqqq]

I decided to look at the implementation, it seems

 try:
    await self._page.evaluate(self._page_script)
 except Exception:
    pass

....
result = await self._page.evaluate("MultimodalWebSurfer.getPageMetadata();")

If any error rises during page script init, MultimodalWebSurfer will be undefined.

Can the team confirm if this is a bug and its cause? I might be able to resolve it.