Stubtesting types which aren't exposed anywhere

[tungol]

I saw #13013 and it reminded me of an idea I have been weighing. Right now stubtest can't run against a type like os._ScanDirIterator because it's not importable. What do people think about providing additional configuration that would help stubtest get access to the runtime class object?

I put together a quick working implementation which goes like this:

Typeshed get a file named stdlib/@tests/stubtest_extra_imports.py. In that file is a dictionary named IMPORT_MAPPING. To get access to os._ScanDirIterator, it looks like this:

import os


def get_ScandirIterator():
    return type(os.scandir("."))


IMPORT_MAPPING = {"os": {"_ScandirIterator": get_ScandirIterator}}

tests/stubtest_stdlib.py adds an extra flag of --imports-file stdlib/@tests/stubtest_imports.py to the stubtest invocation, and stubtest imports this file. When stubtest gets a runtime entry of MISSING, it checks if the entry it's looking for is in the IMPORT_MAPPING table, in which case it executes the function to get the runtime, and then continues testing.

The naming of everything can be tweaked, and this exact arrangement might have trouble scaling. But it works and allows stubtest to produce:

error: os._ScandirIterator cannot be subclassed at runtime, but isn't marked with @final in the stub
Stub: in file /Users/stephen/Developer/clones/typeshed/stdlib/os/__init__.pyi:798
<TypeInfo os._ScandirIterator>
Runtime:
def ()

Thoughts? Is this worth pursuing?

[tungol]

stubtest results found this way:

_codecs._EncodingMap cannot be subclassed at runtime, but isn't marked with @final in the stub
_ctypes._CArgObject cannot be subclassed at runtime, but isn't marked with @final in the stub
_ctypes._CData metaclass mismatch
_ctypes._CData.__release_buffer__ is not present at runtime
_ctypes._CData.from_address is not present at runtime
_ctypes._CData.from_buffer is not present at runtime
_ctypes._CData.from_buffer_copy is not present at runtime
_ctypes._CData.from_param is not present at runtime
_ctypes._CData.in_dll is not present at runtime
_ctypes._CField cannot be subclassed at runtime, but isn't marked with @final in the stub
_ctypes._CField.__get__ is inconsistent, runtime argument "owner" has a default value but stub argument does not
_curses._ncurses_version cannot be subclassed at runtime, but isn't marked with @final in the stub
_curses._ncurses_version._asdict is not present at runtime
_curses._ncurses_version._make is not present at runtime
_curses._ncurses_version._replace is not present at runtime
_curses._ncurses_version.n_fields is not present in stub
_curses._ncurses_version.n_sequence_fields is not present in stub
_curses._ncurses_version.n_unnamed_fields is not present in stub
_dbm._dbm cannot be subclassed at runtime, but isn't marked with @final in the stub
_dbm._dbm.__del__ is not present at runtime
_dbm._dbm.__new__ variable differs from runtime type def (*args: Any, **kwargs: Any) -> Any
_dbm._dbm.get is inconsistent, stub argument "k" should be positional-only (add "/", e.g. "key, /")
_dbm._dbm.get is inconsistent, stub argument "default" should be positional-only (add "/", e.g. "default, /")
_dbm._dbm.setdefault is inconsistent, stub argument "k" should be positional-only (add "/", e.g. "key, /")
_dbm._dbm.setdefault is inconsistent, stub argument "default" should be positional-only (add "/", e.g. "default, /")
_decimal._ContextManager cannot be subclassed at runtime, but isn't marked with @final in the stub
ctypes._CData metaclass mismatch
ctypes._CData.__release_buffer__ is not present at runtime
ctypes._CData.from_address is not present at runtime
ctypes._CData.from_buffer is not present at runtime
ctypes._CData.from_buffer_copy is not present at runtime
ctypes._CData.from_param is not present at runtime
ctypes._CData.in_dll is not present at runtime
curses.panel._Curses_Panel cannot be subclassed at runtime, but isn't marked with @final in the stub
curses.panel._Curses_Panel.move is inconsistent, stub argument "y" should be positional-only (add "/", e.g. "y, /")
curses.panel._Curses_Panel.move is inconsistent, stub argument "x" should be positional-only (add "/", e.g. "x, /")
curses.panel._Curses_Panel.replace is inconsistent, stub argument "win" should be positional-only (add "/", e.g. "win, /")
curses.panel._Curses_Panel.set_userptr is inconsistent, stub argument "obj" should be positional-only (add "/", e.g. "obj, /")
datetime._IsoCalendarDate cannot be subclassed at runtime, but isn't marked with @final in the stub
datetime._IsoCalendarDate._asdict is not present at runtime
datetime._IsoCalendarDate._make is not present at runtime
datetime._IsoCalendarDate._replace is not present at runtime
hashlib._Hash.update is inconsistent, stub argument "data" differs from runtime argument "obj"
hashlib._VarLenHash.digest is inconsistent, stub argument "length" should be positional or keyword (remove "/")
hashlib._VarLenHash.hexdigest is inconsistent, stub argument "length" should be positional or keyword (remove "/")
hashlib._VarLenHash.update is inconsistent, stub argument "data" differs from runtime argument "obj"
os._ScandirIterator cannot be subclassed at runtime, but isn't marked with @final in the stub
sys._asyncgen_hooks.__match_args__ is not present in stub
sys._flags.__match_args__ is not present in stub
sys._flags.int_max_str_digits is not present in stub
sys._float_info.__match_args__ is not present in stub
sys._hash_info.__match_args__ is not present in stub
sys._implementation.__getattr__ is not present at runtime
sys._implementation.__init__ is inconsistent, stub does not have *args argument "args"
sys._int_info.__match_args__ is not present in stub
sys._thread_info.__match_args__ is not present in stub
sys._version_info.__match_args__ is not present in stub
xml.etree.ElementTree._IterParseIterator.__next__ is not a function
xml.etree.ElementTree._IterParseIterator.__next__ is inconsistent, runtime does not have argument "self"
zlib._Compress cannot be subclassed at runtime, but isn't marked with @final in the stub
zlib._Compress.__copy__ is not present in stub
zlib._Compress.__deepcopy__ is not present in stub
zlib._Compress.compress is inconsistent, stub argument "data" should be positional-only (add "/", e.g. "data, /")
zlib._Compress.flush is inconsistent, stub argument "mode" should be positional-only (add "/", e.g. "mode, /")
zlib._Decompress cannot be subclassed at runtime, but isn't marked with @final in the stub
zlib._Decompress.__copy__ is not present in stub
zlib._Decompress.__deepcopy__ is not present in stub
zlib._Decompress.decompress is inconsistent, stub argument "data" should be positional-only (add "/", e.g. "data, /")
zlib._Decompress.flush is inconsistent, stub argument "length" should be positional-only (add "/", e.g. "length, /")