PowLu/TencentOS-tiny
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

micropython: add micropython component

Browse Source
This commit is contained in:
KY-zhang-X
2022-09-29 12:10:37 +08:00
parent 1514f1cb9b
commit dd76146324
2679 changed files with 354110 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

266
components/language/micropython/docs/develop/cmodules.rst Normal file
View File

@@ -0,0 +1,266 @@
.. _cmodules:
MicroPython external C modules
==============================
When developing modules for use with MicroPython you may find you run into
limitations with the Python environment, often due to an inability to access
certain hardware resources or Python speed limitations.
If your limitations can't be resolved with suggestions in :ref:`speed_python`,
writing some or all of your module in C (and/or C++ if implemented for your port)
is a viable option.
If your module is designed to access or work with commonly available
hardware or libraries please consider implementing it inside the MicroPython
source tree alongside similar modules and submitting it as a pull request.
If however you're targeting obscure or proprietary systems it may make
more sense to keep this external to the main MicroPython repository.
This chapter describes how to compile such external modules into the
MicroPython executable or firmware image. Both Make and CMake build
tools are supported, and when writing an external module it's a good idea to
add the build files for both of these tools so the module can be used on all
ports. But when compiling a particular port you will only need to use one
method of building, either Make or CMake.
An alternative approach is to use :ref:`natmod` which allows writing custom C
code that is placed in a .mpy file, which can be imported dynamically in to
a running MicroPython system without the need to recompile the main firmware.
Structure of an external C module
---------------------------------
A MicroPython user C module is a directory with the following files:
* ``*.c`` / ``*.cpp`` / ``*.h`` source code files for your module.
These will typically include the low level functionality being implemented and
the MicroPython binding functions to expose the functions and module(s).
Currently the best reference for writing these functions/modules is
to find similar modules within the MicroPython tree and use them as examples.
* ``micropython.mk`` contains the Makefile fragment for this module.
``$(USERMOD_DIR)`` is available in ``micropython.mk`` as the path to your
module directory. As it's redefined for each c module, is should be expanded
in your ``micropython.mk`` to a local make variable,
eg ``EXAMPLE_MOD_DIR := $(USERMOD_DIR)``
Your ``micropython.mk`` must add your modules source files relative to your
expanded copy of ``$(USERMOD_DIR)`` to ``SRC_USERMOD``, eg
``SRC_USERMOD += $(EXAMPLE_MOD_DIR)/example.c``
If you have custom compiler options (like ``-I`` to add directories to search
for header files), these should be added to ``CFLAGS_USERMOD`` for C code
and to ``CXXFLAGS_USERMOD`` for C++ code.
* ``micropython.cmake`` contains the CMake configuration for this module.
In ``micropython.cmake``, you may use ``${CMAKE_CURRENT_LIST_DIR}`` as the path to
the current module.
Your ``micropython.cmake`` should define an ``INTERFACE`` library and associate
your source files, compile definitions and include directories with it.
The library should then be linked to the ``usermod`` target.
.. code-block:: cmake
add_library(usermod_cexample INTERFACE)
target_sources(usermod_cexample INTERFACE
${CMAKE_CURRENT_LIST_DIR}/examplemodule.c
)
target_include_directories(usermod_cexample INTERFACE
${CMAKE_CURRENT_LIST_DIR}
)
target_link_libraries(usermod INTERFACE usermod_cexample)
See below for full usage example.
Basic example
-------------
This simple module named ``cexample`` provides a single function
``cexample.add_ints(a, b)`` which adds the two integer args together and returns
the result. It can be found in the MicroPython source tree
`in the examples directory <https://github.com/micropython/micropython/tree/master/examples/usercmodule/cexample>`_
and has a source file and a Makefile fragment with content as described above::
micropython/
└──examples/
└──usercmodule/
└──cexample/
├── examplemodule.c
├── micropython.mk
└── micropython.cmake
Refer to the comments in these files for additional explanation.
Next to the ``cexample`` module there's also ``cppexample`` which
works in the same way but shows one way of mixing C and C++ code
in MicroPython.
Compiling the cmodule into MicroPython
--------------------------------------
To build such a module, compile MicroPython (see `getting started
<https://github.com/micropython/micropython/wiki/Getting-Started>`_),
applying 2 modifications:
1. Set the build-time flag ``USER_C_MODULES`` to point to the modules
you want to include. For ports that use Make this variable should be a
directory which is searched automatically for modules. For ports that
use CMake this variable should be a file which includes the modules to
build. See below for details.
2. Enable the modules by setting the corresponding C preprocessor macro to
1. This is only needed if the modules you are building are not
automatically enabled.
For building the example modules which come with MicroPython,
set ``USER_C_MODULES`` to the ``examples/usercmodule`` directory for Make,
or to ``examples/usercmodule/micropython.cmake`` for CMake.
For example, here's how the to build the unix port with the example modules:
.. code-block:: bash
cd micropython/ports/unix
make USER_C_MODULES=../../examples/usercmodule
You may need to run ``make clean`` once at the start when including new
user modules in the build. The build output will show the modules found::
...
Including User C Module from ../../examples/usercmodule/cexample
Including User C Module from ../../examples/usercmodule/cppexample
...
For a CMake-based port such as rp2, this will look a little different (note
that CMake is actually invoked by ``make``):
.. code-block:: bash
cd micropython/ports/rp2
make USER_C_MODULES=../../examples/usercmodule/micropython.cmake
Again, you may need to run ``make clean`` first for CMake to pick up the
user modules. The CMake build output lists the modules by name::
...
Including User C Module(s) from ../../examples/usercmodule/micropython.cmake
Found User C Module(s): usermod_cexample, usermod_cppexample
...
The contents of the top-level ``micropython.cmake`` can be used to control which
modules are enabled.
For your own projects it's more convenient to keep custom code out of the main
MicroPython source tree, so a typical project directory structure will look
like this::
my_project/
├── modules/
│ ├── example1/
│ │ ├── example1.c
│ │ ├── micropython.mk
│ │ └── micropython.cmake
│ ├── example2/
│ │ ├── example2.c
│ │ ├── micropython.mk
│ │ └── micropython.cmake
│ └── micropython.cmake
└── micropython/
├──ports/
... ├──stm32/
...
When building with Make set ``USER_C_MODULES`` to the ``my_project/modules``
directory. For example, building the stm32 port:
.. code-block:: bash
cd my_project/micropython/ports/stm32
make USER_C_MODULES=../../../modules
When building with CMake the top level ``micropython.cmake`` -- found directly
in the ``my_project/modules`` directory -- should ``include`` all of the modules
you want to have available:
.. code-block:: cmake
include(${CMAKE_CURRENT_LIST_DIR}/example1/micropython.cmake)
include(${CMAKE_CURRENT_LIST_DIR}/example2/micropython.cmake)
Then build with:
.. code-block:: bash
cd my_project/micropython/ports/esp32
make USER_C_MODULES=../../../../modules/micropython.cmake
Note that the esp32 port needs the extra ``..`` for relative paths due to the
location of its main ``CMakeLists.txt`` file. You can also specify absolute
paths to ``USER_C_MODULES``.
All modules specified by the ``USER_C_MODULES`` variable (either found in this
directory when using Make, or added via ``include`` when using CMake) will be
compiled, but only those which are enabled will be available for importing.
User modules are usually enabled by default (this is decided by the developer
of the module), in which case there is nothing more to do than set ``USER_C_MODULES``
as described above.
If a module is not enabled by default then the corresponding C preprocessor macro
must be enabled. This macro name can be found by searching for the ``MP_REGISTER_MODULE``
line in the module's source code (it usually appears at the end of the main source file).
This macro should be surrounded by a ``#if X`` / ``#endif`` pair, and the configuration
option ``X`` must be set to 1 using ``CFLAGS_EXTRA`` to make the module available. If
there is no ``#if X`` / ``#endif`` pair then the module is enabled by default.
For example, the ``examples/usercmodule/cexample`` module is enabled by default so
has the following line in its source code:
.. code-block:: c
MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule);
Alternatively, to make this module disabled by default but selectable through
a preprocessor configuration option, it would be:
.. code-block:: c
#if MODULE_CEXAMPLE_ENABLED
MP_REGISTER_MODULE(MP_QSTR_cexample, example_user_cmodule);
#endif
In this case the module is enabled by adding ``CFLAGS_EXTRA=-DMODULE_CEXAMPLE_ENABLED=1``
to the ``make`` command, or editing ``mpconfigport.h`` or ``mpconfigboard.h`` to add
.. code-block:: c
#define MODULE_CEXAMPLE_ENABLED (1)
Note that the exact method depends on the port as they have different
structures. If not done correctly it will compile but importing will
fail to find the module.
Module usage in MicroPython
---------------------------
Once built into your copy of MicroPython, the module
can now be accessed in Python just like any other builtin module, e.g.
.. code-block:: python
import cexample
print(cexample.add_ints(1, 3))
# should display 4

317
components/language/micropython/docs/develop/compiler.rst Normal file
View File

@@ -0,0 +1,317 @@
.. _compiler:
The Compiler
============
The compilation process in MicroPython involves the following steps:
* The lexer converts the stream of text that makes up a MicroPython program into tokens.
* The parser then converts the tokens into an abstract syntax (parse tree).
* Then bytecode or native code is emitted based on the parse tree.
For purposes of this discussion we are going to add a simple language feature ``add1``
that can be use in Python as:
.. code-block:: bash
>>> add1 3
4
>>>
The ``add1`` statement takes an integer as argument and adds ``1`` to it.
Adding a grammar rule
----------------------
MicroPython's grammar is based on the `CPython grammar <https://docs.python.org/3.5/reference/grammar.html>`_
and is defined in `py/grammar.h <https://github.com/micropython/micropython/blob/master/py/grammar.h>`_.
This grammar is what is used to parse MicroPython source files.
There are two macros you need to know to define a grammar rule: ``DEF_RULE`` and ``DEF_RULE_NC``.
``DEF_RULE`` allows you to define a rule with an associated compile function,
while ``DEF_RULE_NC`` has no compile (NC) function for it.
A simple grammar definition with a compile function for our new ``add1`` statement
looks like the following:
.. code-block:: c
DEF_RULE(add1_stmt, c(add1_stmt), and(2), tok(KW_ADD1), rule(testlist))
The second argument ``c(add1_stmt)`` is the corresponding compile function that should be implemented
in ``py/compile.c`` to turn this rule into executable code.
The third required argument can be ``or`` or ``and``. This specifies the number of nodes associated
with a statement. For example, in this case, our ``add1`` statement is similar to ADD1 in assembly
language. It takes one numeric argument. Therefore, the ``add1_stmt`` has two nodes associated with it.
One node is for the statement itself, i.e the literal ``add1`` corresponding to ``KW_ADD1``,
and the other for its argument, a ``testlist`` rule which is the top-level expression rule.
.. note::
The ``add1`` rule here is just an example and not part of the standard
MicroPython grammar.
The fourth argument in this example is the token associated with the rule, ``KW_ADD1``. This token should be
defined in the lexer by editing ``py/lexer.h``.
Defining the same rule without a compile function is achieved by using the ``DEF_RULE_NC`` macro
and omitting the compile function argument:
.. code-block:: c
DEF_RULE_NC(add1_stmt, and(2), tok(KW_ADD1), rule(testlist))
The remaining arguments take on the same meaning. A rule without a compile function must
be handled explicitly by all rules that may have this rule as a node. Such NC-rules are usually
used to express sub-parts of a complicated grammar structure that cannot be expressed in a
single rule.
.. note::
The macros ``DEF_RULE`` and ``DEF_RULE_NC`` take other arguments. For an in-depth understanding of
supported parameters, see `py/grammar.h <https://github.com/micropython/micropython/blob/master/py/grammar.h>`_.
Adding a lexical token
----------------------
Every rule defined in the grammar should have a token associated with it that is defined in ``py/lexer.h``.
Add this token by editing the ``_mp_token_kind_t`` enum:
.. code-block:: c
:emphasize-lines: 12
typedef enum _mp_token_kind_t {
...
MP_TOKEN_KW_OR,
MP_TOKEN_KW_PASS,
MP_TOKEN_KW_RAISE,
MP_TOKEN_KW_RETURN,
MP_TOKEN_KW_TRY,
MP_TOKEN_KW_WHILE,
MP_TOKEN_KW_WITH,
MP_TOKEN_KW_YIELD,
MP_TOKEN_KW_ADD1,
...
} mp_token_kind_t;
Then also edit ``py/lexer.c`` to add the new keyword literal text:
.. code-block:: c
:emphasize-lines: 12
STATIC const char *const tok_kw[] = {
...
"or",
"pass",
"raise",
"return",
"try",
"while",
"with",
"yield",
"add1",
...
};
Notice the keyword is named depending on what you want it to be. For consistency, maintain the
naming standard accordingly.
.. note::
The order of these keywords in ``py/lexer.c`` must match the order of tokens in the enum
defined in ``py/lexer.h``.
Parsing
-------
In the parsing stage the parser takes the tokens produced by the lexer and converts them to an abstract syntax tree (AST) or
*parse tree*. The implementation for the parser is defined in `py/parse.c <https://github.com/micropython/micropython/blob/master/py/parse.c>`_.
The parser also maintains a table of constants for use in different aspects of parsing, similar to what a
`symbol table <https://steemit.com/programming/@drifter1/writing-a-simple-compiler-on-my-own-symbol-table-basic-structure>`_
does.
Several optimizations like `constant folding <http://compileroptimizations.com/category/constant_folding.htm>`_
on integers for most operations e.g. logical, binary, unary, etc, and optimizing enhancements on parenthesis
around expressions are performed during this phase, along with some optimizations on strings.
It's worth noting that *docstrings* are discarded and not accessible to the compiler.
Even optimizations like `string interning <https://en.wikipedia.org/wiki/String_interning>`_ are
not applied to *docstrings*.
Compiler passes
---------------
Like many compilers, MicroPython compiles all code to MicroPython bytecode or native code. The functionality
that achieves this is implemented in `py/compile.c <https://github.com/micropython/micropython/blob/master/py/compile.c>`_.
The most relevant method you should know about is this:
.. code-block:: c
mp_obj_t mp_compile(mp_parse_tree_t *parse_tree, qstr source_file, bool is_repl) {
// Compile the input parse_tree to a raw-code structure.
mp_raw_code_t *rc = mp_compile_to_raw_code(parse_tree, source_file, is_repl);
// Create and return a function object that executes the outer module.
return mp_make_function_from_raw_code(rc, MP_OBJ_NULL, MP_OBJ_NULL);
}
The compiler compiles the code in four passes: scope, stack size, code size and emit.
Each pass runs the same C code over the same AST data structure, with different things
being computed each time based on the results of the previous pass.
First pass
~~~~~~~~~~
In the first pass, the compiler learns about the known identifiers (variables) and
their scope, being global, local, closed over, etc. In the same pass the emitter
(bytecode or native code) also computes the number of labels needed for the emitted
code.
.. code-block:: c
// Compile pass 1.
comp->emit = emit_bc;
comp->emit_method_table = &emit_bc_method_table;
uint max_num_labels = 0;
for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
if (s->emit_options == MP_EMIT_OPT_ASM) {
compile_scope_inline_asm(comp, s, MP_PASS_SCOPE);
} else {
compile_scope(comp, s, MP_PASS_SCOPE);
// Check if any implicitly declared variables should be closed over.
for (size_t i = 0; i < s->id_info_len; ++i) {
id_info_t *id = &s->id_info[i];
if (id->kind == ID_INFO_KIND_GLOBAL_IMPLICIT) {
scope_check_to_close_over(s, id);
}
}
}
...
}
Second and third passes
~~~~~~~~~~~~~~~~~~~~~~~
The second and third passes involve computing the Python stack size and code size
for the bytecode or native code. After the third pass the code size cannot change,
otherwise jump labels will be incorrect.
.. code-block:: c
for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
...
// Pass 2: Compute the Python stack size.
compile_scope(comp, s, MP_PASS_STACK_SIZE);
// Pass 3: Compute the code size.
if (comp->compile_error == MP_OBJ_NULL) {
compile_scope(comp, s, MP_PASS_CODE_SIZE);
}
...
}
Just before pass two there is a selection for the type of code to be emitted, which can
either be native or bytecode.
.. code-block:: c
// Choose the emitter type.
switch (s->emit_options) {
case MP_EMIT_OPT_NATIVE_PYTHON:
case MP_EMIT_OPT_VIPER:
if (emit_native == NULL) {
emit_native = NATIVE_EMITTER(new)(&comp->compile_error, &comp->next_label, max_num_labels);
}
comp->emit_method_table = NATIVE_EMITTER_TABLE;
comp->emit = emit_native;
break;
default:
comp->emit = emit_bc;
comp->emit_method_table = &emit_bc_method_table;
break;
}
The bytecode option is the default but something unique to note for the native
code option is that there is another option via ``VIPER``. See the
:ref:`Emitting native code <emitting_native_code>` section for more details on
viper annotations.
There is also support for *inline assembly code*, where assembly instructions are
written as Python function calls but are emitted directly as the corresponding
machine code. This assembler has only three passes (scope, code size, emit)
and uses a different implementation, not the ``compile_scope`` function.
See the `inline assembler tutorial <https://docs.micropython.org/en/latest/pyboard/tutorial/assembler.html#pyboard-tutorial-assembler>`_
for more details.
Fourth pass
~~~~~~~~~~~
The fourth pass emits the final code that can be executed, either bytecode in
the virtual machine, or native code directly by the CPU.
.. code-block:: c
for (scope_t *s = comp->scope_head; s != NULL && comp->compile_error == MP_OBJ_NULL; s = s->next) {
...
// Pass 4: Emit the compiled bytecode or native code.
if (comp->compile_error == MP_OBJ_NULL) {
compile_scope(comp, s, MP_PASS_EMIT);
}
}
Emitting bytecode
-----------------
Statements in Python code usually correspond to emitted bytecode, for example ``a + b``
generates "push a" then "push b" then "binary op add". Some statements do not emit
anything but instead affect other things like the scope of variables, for example
``global a``.
The implementation of a function that emits bytecode looks similar to this:
.. code-block:: c
void mp_emit_bc_unary_op(emit_t *emit, mp_unary_op_t op) {
emit_write_bytecode_byte(emit, 0, MP_BC_UNARY_OP_MULTI + op);
}
We use the unary operator expressions for an example here but the implementation
details are similar for other statements/expressions. The method ``emit_write_bytecode_byte()``
is a wrapper around the main function ``emit_get_cur_to_write_bytecode()`` that all
functions must call to emit bytecode.
.. _emitting_native_code:
Emitting native code
---------------------
Similar to how bytecode is generated, there should be a corresponding function in ``py/emitnative.c`` for each
code statement:
.. code-block:: c
STATIC void emit_native_unary_op(emit_t *emit, mp_unary_op_t op) {
vtype_kind_t vtype;
emit_pre_pop_reg(emit, &vtype, REG_ARG_2);
if (vtype == VTYPE_PYOBJ) {
emit_call_with_imm_arg(emit, MP_F_UNARY_OP, op, REG_ARG_1);
emit_post_push_reg(emit, VTYPE_PYOBJ, REG_RET);
} else {
adjust_stack(emit, 1);
EMIT_NATIVE_VIPER_TYPE_ERROR(emit,
MP_ERROR_TEXT("unary op %q not implemented"), mp_unary_op_method_name[op]);
}
}
The difference here is that we have to handle *viper typing*. Viper annotations allow
us to handle more than one type of variable. By default all variables are Python objects,
but with viper a variable can also be declared as a machine-typed variable like a native
integer or pointer. Viper can be thought of as a superset of Python, where normal Python
objects are handled as usual, while native machine variables are handled in an optimised
way by using direct machine instructions for the operations. Viper typing may break
Python equivalence because, for example, integers become native integers and can overflow
(unlike Python integers which extend automatically to arbitrary precision).

18
components/language/micropython/docs/develop/extendingmicropython.rst Normal file
View File

@@ -0,0 +1,18 @@
.. _extendingmicropython:
Extending MicroPython in C
==========================
This chapter describes options for implementing additional functionality in C, but from code
written outside of the main MicroPython repository. The first approach is useful for building
your own custom firmware with some project-specific additional modules or functions that can
be accessed from Python. The second approach is for building modules that can be loaded at runtime.
Please see the :ref:`library section <internals_library>` for more information on building core modules that
live in the main MicroPython repository.
.. toctree::
:maxdepth: 3
cmodules.rst
natmod.rst

329
components/language/micropython/docs/develop/gettingstarted.rst Normal file
View File

@@ -0,0 +1,329 @@
.. _gettingstarted:
Getting Started
===============
This guide covers a step-by-step process on setting up version control, obtaining and building
a copy of the source code for a port, building the documentation, running tests, and a description of the
directory structure of the MicroPython code base.
Source control with git
-----------------------
MicroPython is hosted on `GitHub <https://github.com/micropython/micropython>`_ and uses
`Git <https://git-scm.com>`_ for source control. The workflow is such that
code is pulled and pushed to and from the main repository. Install the respective version
of Git for your operating system to follow through the rest of the steps.
.. note::
For a reference on the installation instructions, please refer to
the `Git installation instructions <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`_.
Learn about the basic git commands in this `Git Handbook <https://guides.github.com/introduction/git-handbook/>`_
or any other sources on the internet.
.. note::
A .git-blame-ignore-revs file is included which avoids the output of git blame getting cluttered
by commits which are only for formatting code but have no functional changes. See `git blame documentation
<https://git-scm.com/docs/git-blame#Documentation/git-blame.txt---ignore-revltrevgt>`_ on how to use this.
Get the code
------------
It is recommended that you maintain a fork of the MicroPython repository for your development purposes.
The process of obtaining the source code includes the following:
#. Fork the repository https://github.com/micropython/micropython
#. You will now have a fork at <https://github.com/<your-user-name>/micropython>.
#. Clone the forked repository using the following command:
.. code-block:: bash
$ git clone https://github.com/<your-user-name>/micropython
Then, `configure the remote repositories <https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes>`_ to be able to
collaborate on the MicroPython project.
Configure remote upstream:
.. code-block:: bash
$ cd micropython
$ git remote add upstream https://github.com/micropython/micropython
It is common to configure ``upstream`` and ``origin`` on a forked repository
to assist with sharing code changes. You can maintain your own mapping but
it is recommended that ``origin`` maps to your fork and ``upstream`` to the main
MicroPython repository.
After the above configuration, your setup should be similar to this:
.. code-block:: bash
$ git remote -v
origin https://github.com/<your-user-name>/micropython (fetch)
origin https://github.com/<your-user-name>/micropython (push)
upstream https://github.com/micropython/micropython (fetch)
upstream https://github.com/micropython/micropython (push)
You should now have a copy of the source code. By default, you are pointing
to the master branch. To prepare for further development, it is recommended
to work on a development branch.
.. code-block:: bash
$ git checkout -b dev-branch
You can give it any name. You will have to compile MicroPython whenever you change
to a different branch.
Compile and build the code
--------------------------
When compiling MicroPython, you compile a specific :term:`port`, usually
targeting a specific :ref:`board <glossary>`. Start by installing the required dependencies.
Then build the MicroPython cross-compiler before you can successfully compile and build.
This applies specifically when using Linux to compile.
The Windows instructions are provided in a later section.
.. _required_dependencies:
Required dependencies
~~~~~~~~~~~~~~~~~~~~~
Install the required dependencies for Linux:
.. code-block:: bash
$ sudo apt-get install build-essential libffi-dev git pkg-config
For the stm32 port, the ARM cross-compiler is required:
.. code-block:: bash
$ sudo apt-get install arm-none-eabi-gcc arm-none-eabi-binutils arm-none-eabi-newlib
See the `ARM GCC
toolchain <https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm>`_
for the latest details.
Python is also required. Python 2 is supported for now, but we recommend using Python 3.
Check that you have Python available on your system:
.. code-block:: bash
$ python3
Python 3.5.0 (default, Jul 17 2020, 14:04:10)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
All supported ports have different dependency requirements, see their respective
`readme files <https://github.com/micropython/micropython/tree/master/ports>`_.
Building the MicroPython cross-compiler
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Almost all ports require building ``mpy-cross`` first to perform pre-compilation
of Python code that will be included in the port firmware:
.. code-block:: bash
$ cd mpy-cross
$ make
.. note::
Note that, ``mpy-cross`` must be built for the host architecture
and not the target architecture.
If it built successfully, you should see a message similar to this:
.. code-block:: bash
LINK mpy-cross
text data bss dec hex filename
279328 776 880 280984 44998 mpy-cross
.. note::
Use ``make -C mpy-cross`` to build the cross-compiler in one statement
without moving to the ``mpy-cross`` directory otherwise, you will need
to do ``cd ..`` for the next steps.
Building the Unix port of MicroPython
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Unix port is a version of MicroPython that runs on Linux, macOS, and other Unix-like operating systems.
It's extremely useful for developing MicroPython as it avoids having to deploy your code to a device to test it.
In many ways, it works a lot like CPython's python binary.
To build for the Unix port, make sure all Linux related dependencies are installed as detailed in the
required dependencies section. See the :ref:`required_dependencies`
to make sure that all dependencies are installed for this port. Also, make sure you have a working
environment for ``gcc`` and ``GNU make``. Ubuntu 20.04 has been used for the example
below but other unixes ought to work with little modification:
.. code-block:: bash
$ gcc --version
gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.then build:
.. code-block:: bash
$ cd ports/unix
$ make submodules
$ make
If MicroPython built correctly, you should see the following:
.. code-block:: bash
LINK micropython
text data bss dec hex filename
412033 5680 2496 420209 66971 micropython
Now run it:
.. code-block:: bash
$ ./micropython
MicroPython v1.13-38-gc67012d-dirty on 2020-09-13; linux version
Use Ctrl-D to exit, Ctrl-E for paste mode
>>> print("hello world")
hello world
>>>
Building the Windows port
~~~~~~~~~~~~~~~~~~~~~~~~~
The Windows port includes a Visual Studio project file micropython.vcxproj that you can use to build micropython.exe.
It can be opened in Visual Studio or built from the command line using msbuild. Alternatively, it can be built using mingw,
either in Windows with Cygwin, or on Linux.
See `windows port documentation <https://github.com/micropython/micropython/tree/master/ports/windows>`_ for more information.
Building the STM32 port
~~~~~~~~~~~~~~~~~~~~~~~
Like the Unix port, you need to install some required dependencies
as detailed in the :ref:`required_dependencies` section, then build:
.. code-block:: bash
$ cd ports/stm32
$ make submodules
$ make
Please refer to the `stm32 documentation <https://github.com/micropython/micropython/tree/master/ports/stm32>`_
for more details on flashing the firmware.
.. note::
See the :ref:`required_dependencies` to make sure that all dependencies are installed for this port.
The cross-compiler is needed. ``arm-none-eabi-gcc`` should also be in the $PATH or specified manually
via CROSS_COMPILE, either by setting the environment variable or in the ``make`` command line arguments.
You can also specify which board to use:
.. code-block:: bash
$ cd ports/stm32
$ make submodules
$ make BOARD=<board>
See `ports/stm32/boards <https://github.com/micropython/micropython/tree/master/ports/stm32/boards>`_
for the available boards. e.g. "PYBV11" or "NUCLEO_WB55".
Building the documentation
--------------------------
MicroPython documentation is created using ``Sphinx``. If you have already
installed Python, then install ``Sphinx`` using ``pip``. It is recommended
that you use a virtual environment:
.. code-block:: bash
$ python3 -m venv env
$ source env/bin/activate
$ pip install sphinx
Navigate to the ``docs`` directory:
.. code-block:: bash
$ cd docs
Build the docs:
.. code-block:: bash
$ make html
Open ``docs/build/html/index.html`` in your browser to view the docs locally. Refer to the
documentation on `importing your documentation
<https://docs.readthedocs.io/en/stable/intro/import-guide.html>`_ to use Read the Docs.
Running the tests
-----------------
To run all tests in the test suite on the Unix port use:
.. code-block:: bash
$ cd ports/unix
$ make test
To run a selection of tests on a board/device connected over USB use:
.. code-block:: bash
$ cd tests
$ ./run-tests.py --target minimal --device /dev/ttyACM0
See also :ref:`writingtests`.
Folder structure
----------------
There are a couple of directories to take note of in terms of where certain implementation details
are. The following is a break down of the top-level folders in the source code.
py
Contains the compiler, runtime, and core library implementation.
mpy-cross
Has the MicroPython cross-compiler which pre-compiles the Python scripts to bytecode.
ports
Code for all the versions of MicroPython for the supported ports.
lib
Low-level C libraries used by any port which are mostly 3rd-party libraries.
drivers
Has drivers for specific hardware and intended to work across multiple ports.
extmod
Contains a C implementation of more non-core modules.
docs
Has the standard documentation found at https://docs.micropython.org/.
tests
An implementation of the test suite.
tools
Contains helper tools including the ``upip`` and the ``pyboard.py`` module.
examples
Example code for building MicroPython as a library as well as native modules.

BIN
components/language/micropython/docs/develop/img/bitmap.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 6.2 KiB

BIN
components/language/micropython/docs/develop/img/collision.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.4 KiB

BIN
components/language/micropython/docs/develop/img/linprob.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.6 KiB

26
components/language/micropython/docs/develop/index.rst Normal file
View File

@@ -0,0 +1,26 @@
MicroPython Internals
=====================
This chapter covers a tour of MicroPython from the perspective of a developer, contributing
to MicroPython. It acts as a comprehensive resource on the implementation details of MicroPython
for both novice and expert contributors.
Development around MicroPython usually involves modifying the core runtime, porting or
maintaining a new library. This guide describes at great depth, the implementation
details of MicroPython including a getting started guide, compiler internals, porting
MicroPython to a new platform and implementing a core MicroPython library.
.. toctree::
:maxdepth: 3
gettingstarted.rst
writingtests.rst
compiler.rst
memorymgt.rst
library.rst
optimizations.rst
qstr.rst
maps.rst
publiccapi.rst
extendingmicropython.rst
porting.rst

86
components/language/micropython/docs/develop/library.rst Normal file
View File

@@ -0,0 +1,86 @@
.. _internals_library:
Implementing a Module
=====================
This chapter details how to implement a core module in MicroPython.
MicroPython modules can be one of the following:
- Built-in module: A general module that is be part of the MicroPython repository.
- User module: A module that is useful for your specific project that you maintain
in your own repository or private codebase.
- Dynamic module: A module that can be deployed and imported at runtime to your device.
A module in MicroPython can be implemented in one of the following locations:
- py/: A core library that mirrors core CPython functionality.
- extmod/: A CPython or MicroPython-specific module that is shared across multiple ports.
- ports/<port>/: A port-specific module.
.. note::
This chapter describes modules implemented in ``py/`` or core modules.
See :ref:`extendingmicropython` for details on implementing an external module.
For details on port-specific modules, see :ref:`porting_to_a_board`.
Implementing a core module
--------------------------
Like CPython, MicroPython has core builtin modules that can be accessed through import statements.
An example is the ``gc`` module discussed in :ref:`memorymanagement`.
.. code-block:: bash
>>> import gc
>>> gc.enable()
>>>
MicroPython has several other builtin standard/core modules like ``io``, ``array`` etc.
Adding a new core module involves several modifications.
First, create the ``C`` file in the ``py/`` directory. In this example we are adding a
hypothetical new module ``subsystem`` in the file ``modsubsystem.c``:
.. code-block:: c
#include "py/builtin.h"
#include "py/runtime.h"
#if MICROPY_PY_SUBSYSTEM
// info()
STATIC mp_obj_t py_subsystem_info(void) {
return MP_OBJ_NEW_SMALL_INT(42);
}
MP_DEFINE_CONST_FUN_OBJ_0(subsystem_info_obj, py_subsystem_info);
STATIC const mp_rom_map_elem_t mp_module_subsystem_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_subsystem) },
{ MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&subsystem_info_obj) },
};
STATIC MP_DEFINE_CONST_DICT(mp_module_subsystem_globals, mp_module_subsystem_globals_table);
const mp_obj_module_t mp_module_subsystem = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t *)&mp_module_subsystem_globals,
};
MP_REGISTER_MODULE(MP_QSTR_subsystem, mp_module_subsystem);
#endif
The implementation includes a definition of all functions related to the module and adds the
functions to the module's global table in ``mp_module_subsystem_globals_table``. It also
creates the module object with ``mp_module_subsystem``. The module is then registered with
the wider system via the ``MP_REGISTER_MODULE`` macro.
After building and running the modified MicroPython, the module should now be importable:
.. code-block:: bash
>>> import subsystem
>>> subsystem.info()
42
>>>
Our ``info()`` function currently returns just a single number but can be extended
to do anything. Similarly, more functions can be added to this new module.

63
components/language/micropython/docs/develop/maps.rst Normal file
View File

@@ -0,0 +1,63 @@
.. _maps:
Maps and Dictionaries
=====================
MicroPython dictionaries and maps use techniques called open addressing and linear probing.
This chapter details both of these methods.
Open addressing
---------------
`Open addressing <https://en.wikipedia.org/wiki/Open_addressing>`_ is used to resolve collisions.
Collisions are very common occurrences and happen when two items happen to hash to the same
slot or location. For example, given a hash setup as this:
.. image:: img/collision.png
If there is a request to fill slot ``0`` with ``70``, since the slot ``0`` is not empty, open addressing
finds the next available slot in the dictionary to service this request. This sequential search for an alternate
location is called *probing*. There are several sequence probing algorithms but MicroPython uses
linear probing that is described in the next section.
Linear probing
--------------
Linear probing is one of the methods for finding an available address or slot in a dictionary. In MicroPython,
it is used with open addressing. To service the request described above, unlike other probing algorithms,
linear probing assumes a fixed interval of ``1`` between probes. The request will therefore be serviced by
placing the item in the next free slot which is slot ``4`` in our example:
.. image:: img/linprob.png
The same methods i.e open addressing and linear probing are used to search for an item in a dictionary.
Assume we want to search for the data item ``33``. The computed hash value will be 2. Looking at slot 2
reveals ``33``, at this point, we return ``True``. Searching for ``70`` is quite different as there was a
collision at the time of insertion. Therefore computing the hash value is ``0`` which is currently
holding ``44``. Instead of simply returning ``False``, we perform a sequential search starting at point
``1`` until the item ``70`` is found or we encounter a free slot. This is the general way of performing
look-ups in hashes:
.. code-block:: c
// not yet found, keep searching in this table
pos = (pos + 1) % set->alloc;
if (pos == start_pos) {
// search got back to starting position, so index is not in table
if (lookup_kind & MP_MAP_LOOKUP_ADD_IF_NOT_FOUND) {
if (avail_slot != NULL) {
// there was an available slot, so use that
set->used++;
*avail_slot = index;
return index;
} else {
// not enough room in table, rehash it
mp_set_rehash(set);
// restart the search for the new element
start_pos = pos = hash % set->alloc;
}
}
} else {
return MP_OBJ_NULL;
}

141
components/language/micropython/docs/develop/memorymgt.rst Normal file
View File

@@ -0,0 +1,141 @@
.. _memorymanagement:
Memory Management
=================
Unlike programming languages such as C/C++, MicroPython hides memory management
details from the developer by supporting automatic memory management.
Automatic memory management is a technique used by operating systems or applications to automatically manage
the allocation and deallocation of memory. This eliminates challenges such as forgetting to
free the memory allocated to an object. Automatic memory management also avoids the critical issue of using memory
that is already released. Automatic memory management takes many forms, one of them being
garbage collection (GC).
The garbage collector usually has two responsibilities;
#. Allocate new objects in available memory.
#. Free unused memory.
There are many GC algorithms but MicroPython uses the
`Mark and Sweep <https://en.wikipedia.org/wiki/Tracing_garbage_collection#Basic_algorithm>`_
policy for managing memory. This algorithm has a mark phase that traverses the heap marking all
live objects while the sweep phase goes through the heap reclaiming all unmarked objects.
Garbage collection functionality in MicroPython is available through the ``gc`` built-in
module:
.. code-block:: bash
>>> x = 5
>>> x
5
>>> import gc
>>> gc.enable()
>>> gc.mem_alloc()
1312
>>> gc.mem_free()
2071392
>>> gc.collect()
19
>>> gc.disable()
>>>
Even when ``gc.disable()`` is invoked, collection can be triggered with ``gc.collect()``.
The object model
----------------
All MicroPython objects are referred to by the ``mp_obj_t`` data type.
This is usually word-sized (i.e. the same size as a pointer on the target architecture),
and can be typically 32-bit (STM32, nRF, ESP32, Unix x86) or 64-bit (Unix x64).
It can also be greater than a word-size for certain object representations, for
example ``OBJ_REPR_D`` has a 64-bit sized ``mp_obj_t`` on a 32-bit architecture.
An ``mp_obj_t`` represents a MicroPython object, for example an integer, float, type, dict or
class instance. Some objects, like booleans and small integers, have their value stored directly
in the ``mp_obj_t`` value and do not require additional memory. Other objects have their value
store elsewhere in memory (for example on the garbage-collected heap) and their ``mp_obj_t`` contains
a pointer to that memory. A portion of ``mp_obj_t`` is the tag which tells what type of object it is.
See ``py/mpconfig.h`` for the specific details of the available representations.
**Pointer tagging**
Because pointers are word-aligned, when they are stored in an ``mp_obj_t`` the
lower bits of this object handle will be zero. For example on a 32-bit architecture
the lower 2 bits will be zero:
``********|********|********|******00``
These bits are reserved for purposes of storing a tag. The tag stores extra information as
opposed to introducing a new field to store that information in the object, which may be
inefficient. In MicroPython the tag tells if we are dealing with a small integer, interned
(small) string or a concrete object, and different semantics apply to each of these.
For small integers the mapping is this:
``********|********|********|*******1``
Where the asterisks hold the actual integer value. For an interned string or an immediate
object (e.g. ``True``) the layout of the ``mp_obj_t`` value is, respectively:
``********|********|********|*****010``
``********|********|********|*****110``
While a concrete object that is none of the above takes the form:
``********|********|********|******00``
The stars here correspond to the address of the concrete object in memory.
Allocation of objects
----------------------
The value of a small integer is stored directly in the ``mp_obj_t`` and will be
allocated in-place, not on the heap or elsewhere. As such, creation of small
integers does not affect the heap. Similarly for interned strings that already have
their textual data stored elsewhere, and immediate values like ``None``, ``False``
and ``True``.
Everything else which is a concrete object is allocated on the heap and its object structure is such that
a field is reserved in the object header to store the type of the object.
.. code-block:: bash
+++++++++++
+ +
+ type + object header
+ +
+++++++++++
+ + object items
+ +
+ +
+++++++++++
The heap's smallest unit of allocation is a block, which is four machine words in
size (16 bytes on a 32-bit machine, 32 bytes on a 64-bit machine).
Another structure also allocated on the heap tracks the allocation of
objects in each block. This structure is called a *bitmap*.
.. image:: img/bitmap.png
The bitmap tracks whether a block is "free" or "in use" and use two bits to track this state
for each block.
The mark-sweep garbage collector manages the objects allocated on the heap, and also
utilises the bitmap to mark objects that are still in use.
See `py/gc.c <https://github.com/micropython/micropython/blob/master/py/gc.c>`_
for the full implementation of these details.
**Allocation: heap layout**
The heap is arranged such that it consists of blocks in pools. A block
can have different properties:
- *ATB(allocation table byte):* If set, then the block is a normal block
- *FREE:* Free block
- *HEAD:* Head of a chain of blocks
- *TAIL:* In the tail of a chain of blocks
- *MARK :* Marked head block
- *FTB(finaliser table byte):* If set, then the block has a finaliser

225
components/language/micropython/docs/develop/natmod.rst Normal file
View File

@@ -0,0 +1,225 @@
.. _natmod:
Native machine code in .mpy files
=================================
This section describes how to build and work with .mpy files that contain native
machine code from a language other than Python. This allows you to
write code in a language like C, compile and link it into a .mpy file, and then
import this file like a normal Python module. This can be used for implementing
functionality which is performance critical, or for including an existing
library written in another language.
One of the main advantages of using native .mpy files is that native machine code
can be imported by a script dynamically, without the need to rebuild the main
MicroPython firmware. This is in contrast to :ref:`cmodules` which also allows
defining custom modules in C but they must be compiled into the main firmware image.
The focus here is on using C to build native modules, but in principle any
language which can be compiled to stand-alone machine code can be put into a
.mpy file.
A native .mpy module is built using the ``mpy_ld.py`` tool, which is found in the
``tools/`` directory of the project. This tool takes a set of object files
(.o files) and links them together to create a native .mpy files. It requires
CPython 3 and the library pyelftools v0.25 or greater.
Supported features and limitations
----------------------------------
A .mpy file can contain MicroPython bytecode and/or native machine code. If it
contains native machine code then the .mpy file has a specific architecture
associated with it. Current supported architectures are (these are the valid
options for the ``ARCH`` variable, see below):
* ``x86`` (32 bit)
* ``x64`` (64 bit x86)
* ``armv6m`` (ARM Thumb, eg Cortex-M0)
* ``armv7m`` (ARM Thumb 2, eg Cortex-M3)
* ``armv7emsp`` (ARM Thumb 2, single precision float, eg Cortex-M4F, Cortex-M7)
* ``armv7emdp`` (ARM Thumb 2, double precision float, eg Cortex-M7)
* ``xtensa`` (non-windowed, eg ESP8266)
* ``xtensawin`` (windowed with window size 8, eg ESP32)
When compiling and linking the native .mpy file the architecture must be chosen
and the corresponding file can only be imported on that architecture. For more
details about .mpy files see :ref:`mpy_files`.
Native code must be compiled as position independent code (PIC) and use a global
offset table (GOT), although the details of this varies from architecture to
architecture. When importing .mpy files with native code the import machinery
is able to do some basic relocation of the native code. This includes
relocating text, rodata and BSS sections.
Supported features of the linker and dynamic loader are:
* executable code (text)
* read-only data (rodata), including strings and constant data (arrays, structs, etc)
* zeroed data (BSS)
* pointers in text to text, rodata and BSS
* pointers in rodata to text, rodata and BSS
The known limitations are:
* data sections are not supported; workaround: use BSS data and initialise the
data values explicitly
* static BSS variables are not supported; workaround: use global BSS variables
So, if your C code has writable data, make sure the data is defined globally,
without an initialiser, and only written to within functions.
Linker limitation: the native module is not linked against the symbol table of the
full MicroPython firmware. Rather, it is linked against an explicit table of exported
symbols found in ``mp_fun_table`` (in ``py/nativeglue.h``), that is fixed at firmware
build time. It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system
function, for example.
New symbols can be added to the end of the table and the firmware rebuilt.
The symbols also need to be added to ``tools/mpy_ld.py``'s ``fun_table`` dict in the
same location. This allows ``mpy_ld.py`` to be able to pick the new symbols up and
provide relocations for them when the mpy is imported. Finally, if the symbol is a
function, a macro or stub should be added to ``py/dynruntime.h`` to make it easy to
call the function.
Defining a native module
------------------------
A native .mpy module is defined by a set of files that are used to build the .mpy.
The filesystem layout consists of two main parts, the source files and the Makefile:
* In the simplest case only a single C source file is required, which contains all
the code that will be compiled into the .mpy module. This C source code must
include the ``py/dynruntime.h`` file to access the MicroPython dynamic API, and
must at least define a function called ``mpy_init``. This function will be the
entry point of the module, called when the module is imported.
The module can be split into multiple C source files if desired. Parts of the
module can also be implemented in Python. All source files should be listed in
the Makefile, by adding them to the ``SRC`` variable (see below). This includes
both C source files as well as any Python files which will be included in the
resulting .mpy file.
* The ``Makefile`` contains the build configuration for the module and list the
source files used to build the .mpy module. It should define ``MPY_DIR`` as the
location of the MicroPython repository (to find header files, the relevant Makefile
fragment, and the ``mpy_ld.py`` tool), ``MOD`` as the name of the module, ``SRC``
as the list of source files, optionally specify the machine architecture via ``ARCH``,
and then include ``py/dynruntime.mk``.
Minimal example
---------------
This section provides a fully working example of a simple module named ``factorial``.
This module provides a single function ``factorial.factorial(x)`` which computes the
factorial of the input and returns the result.
Directory layout::
factorial/
├── factorial.c
└── Makefile
The file ``factorial.c`` contains:
.. code-block:: c
// Include the header file to get access to the MicroPython API
#include "py/dynruntime.h"
// Helper function to compute factorial
STATIC mp_int_t factorial_helper(mp_int_t x) {
if (x == 0) {
return 1;
}
return x * factorial_helper(x - 1);
}
// This is the function which will be called from Python, as factorial(x)
STATIC mp_obj_t factorial(mp_obj_t x_obj) {
// Extract the integer from the MicroPython input object
mp_int_t x = mp_obj_get_int(x_obj);
// Calculate the factorial
mp_int_t result = factorial_helper(x);
// Convert the result to a MicroPython integer object and return it
return mp_obj_new_int(result);
}
// Define a Python reference to the function above
STATIC MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial);
// This is the entry point and is called when the module is imported
mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) {
// This must be first, it sets up the globals dict and other things
MP_DYNRUNTIME_INIT_ENTRY
// Make the function available in the module's namespace
mp_store_global(MP_QSTR_factorial, MP_OBJ_FROM_PTR(&factorial_obj));
// This must be last, it restores the globals dict
MP_DYNRUNTIME_INIT_EXIT
}
The file ``Makefile`` contains:
.. code-block:: make
# Location of top-level MicroPython directory
MPY_DIR = ../../..
# Name of module
MOD = factorial
# Source files (.c or .py)
SRC = factorial.c
# Architecture to build for (x86, x64, armv6m, armv7m, xtensa, xtensawin)
ARCH = x64
# Include to get the rules for compiling and linking the module
include $(MPY_DIR)/py/dynruntime.mk
Compiling the module
--------------------
The prerequisite tools needed to build a native .mpy file are:
* The MicroPython repository (at least the ``py/`` and ``tools/`` directories).
* CPython 3, and the library pyelftools (eg ``pip install 'pyelftools>=0.25'``).
* GNU make.
* A C compiler for the target architecture (if C source is used).
* Optionally ``mpy-cross``, built from the MicroPython repository (if .py source is used).
Be sure to select the correct ``ARCH`` for the target you are going to run on.
Then build with::
$ make
Without modifying the Makefile you can specify the target architecture via::
$ make ARCH=armv7m
Module usage in MicroPython
---------------------------
Once the module is built there should be a file called ``factorial.mpy``. Copy
this so it is accessible on the filesystem of your MicroPython system and can be
found in the import path. The module can now be accessed in Python just like any
other module, for example::
import factorial
print(factorial.factorial(10))
# should display 3628800
Further examples
----------------
See ``examples/natmod/`` for further examples which show many of the available
features of native .mpy modules. Such features include:
* using multiple C source files
* including Python code alongside C code
* rodata and BSS data
* memory allocation
* use of floating point
* exception handling
* including external C libraries

72
components/language/micropython/docs/develop/optimizations.rst Normal file
View File

@@ -0,0 +1,72 @@
.. _optimizations:
Optimizations
=============
MicroPython uses several optimizations to save RAM but also ensure the efficient
execution of programs. This chapter discusses some of these optimizations.
.. note::
:ref:`qstr` and :ref:`maps` details other optimizations on strings and
dictionaries.
Frozen bytecode
---------------
When MicroPython loads Python code from the filesystem, it first has to parse the file into
a temporary in-memory representation, and then generate bytecode for execution, both of which
are stored in the heap (in RAM). This can lead to significant amounts of memory being used.
The MicroPython cross compiler can be used to generate
a ``.mpy`` file, containing the pre-compiled bytecode for a Python module. This will still
be loaded into RAM, but it avoids the additional overhead of the parsing stage.
As a further optimisation, the pre-compiled bytecode from a ``.mpy`` file can be "frozen"
into the firmware image as part of the main firmware compilation process, which means that
the bytecode will be executed from ROM. This can lead to a significant memory saving, and
reduce heap fragmentation.
Variables
---------
MicroPython processes local and global variables differently. Global variables
are stored and looked up from a global dictionary that is allocated on the heap
(note that each module has its own separate dict, so separate namespace).
Local variables on the other hand are are stored on the Python value stack, which may
live on the C stack or on the heap. They are accessed directly by their offset
within the Python stack, which is more efficient than a global lookup in a dict.
The length of global variable names also affects how much RAM is used as identifiers
are stored in RAM. The shorter the identifier, the less memory is used.
The other aspect is that ``const`` variables that start with an underscore are treated as
proper constants and are not allocated or added in a dictionary, hence saving some memory.
These variables use ``const()`` from the MicroPython library. Therefore:
.. code-block:: python
from micropython import const
X = const(1)
_Y = const(2)
foo(X, _Y)
Compiles to:
.. code-block:: python
X = 1
foo(1, 2)
Allocation of memory
--------------------
Most of the common MicroPython constructs are not allocated on the heap.
However the following are:
- Dynamic data structures like lists, mappings, etc;
- Functions, classes and object instances;
- imports; and
- First-time assignment of global variables (to create the slot in the global dict).
For a detailed discussion on a more user-centric perspective on optimization,
see `Maximising MicroPython speed <https://docs.micropython.org/en/latest/reference/speed_python.html>`_

306
components/language/micropython/docs/develop/porting.rst Normal file
View File

@@ -0,0 +1,306 @@
.. _porting_to_a_board:
Porting MicroPython
===================
The MicroPython project contains several ports to different microcontroller families and
architectures. The project repository has a `ports <https://github.com/micropython/micropython/tree/master/ports>`_
directory containing a subdirectory for each supported port.
A port will typically contain definitions for multiple "boards", each of which is a specific piece of
hardware that that port can run on, e.g. a development kit or device.
The `minimal port <https://github.com/micropython/micropython/tree/master/ports/minimal>`_ is
available as a simplified reference implementation of a MicroPython port. It can run on both the
host system and an STM32F4xx MCU.
In general, starting a port requires:
- Setting up the toolchain (configuring Makefiles, etc).
- Implementing boot configuration and CPU initialization.
- Initialising basic drivers required for development and debugging (e.g. GPIO, UART).
- Performing the board-specific configurations.
- Implementing the port-specific modules.
Minimal MicroPython firmware
----------------------------
The best way to start porting MicroPython to a new board is by integrating a minimal
MicroPython interpreter. For this walkthrough, create a subdirectory for the new
port in the ``ports`` directory:
.. code-block:: bash
$ cd ports
$ mkdir example_port
The basic MicroPython firmware is implemented in the main port file, e.g ``main.c``:
.. code-block:: c
#include "py/compile.h"
#include "py/gc.h"
#include "py/mperrno.h"
#include "py/stackctrl.h"
#include "shared/runtime/gchelper.h"
#include "shared/runtime/pyexec.h"
// Allocate memory for the MicroPython GC heap.
static char heap[4096];
int main(int argc, char **argv) {
// Initialise the MicroPython runtime.
mp_stack_ctrl_init();
gc_init(heap, heap + sizeof(heap));
mp_init();
// Start a normal REPL; will exit when ctrl-D is entered on a blank line.
pyexec_friendly_repl();
// Deinitialise the runtime.
gc_sweep_all();
mp_deinit();
return 0;
}
// Handle uncaught exceptions (should never be reached in a correct C implementation).
void nlr_jump_fail(void *val) {
for (;;) {
}
}
// Do a garbage collection cycle.
void gc_collect(void) {
gc_collect_start();
gc_helper_collect_regs_and_stack();
gc_collect_end();
}
// There is no filesystem so stat'ing returns nothing.
mp_import_stat_t mp_import_stat(const char *path) {
return MP_IMPORT_STAT_NO_EXIST;
}
// There is no filesystem so opening a file raises an exception.
mp_lexer_t *mp_lexer_new_from_file(const char *filename) {
mp_raise_OSError(MP_ENOENT);
}
We also need a Makefile at this point for the port:
.. code-block:: Makefile
# Include the core environment definitions; this will set $(TOP).
include ../../py/mkenv.mk
# Include py core make definitions.
include $(TOP)/py/py.mk
# Set CFLAGS and libraries.
CFLAGS = -I. -I$(BUILD) -I$(TOP)
LIBS = -lm
# Define the required source files.
SRC_C = \
main.c \
mphalport.c \
shared/readline/readline.c \
shared/runtime/gchelper_generic.c \
shared/runtime/pyexec.c \
shared/runtime/stdout_helpers.c \
# Define the required object files.
OBJ = $(PY_CORE_O) $(addprefix $(BUILD)/, $(SRC_C:.c=.o))
# Define the top-level target, the main firmware.
all: $(BUILD)/firmware.elf
# Define how to build the firmware.
$(BUILD)/firmware.elf: $(OBJ)
$(ECHO) "LINK $@"
$(Q)$(CC) $(LDFLAGS) -o $@ $^ $(LIBS)
$(Q)$(SIZE) $@
# Include remaining core make rules.
include $(TOP)/py/mkrules.mk
Remember to use proper tabs to indent the Makefile.
MicroPython Configurations
--------------------------
After integrating the minimal code above, the next step is to create the MicroPython
configuration files for the port. The compile-time configurations are specified in
``mpconfigport.h`` and additional hardware-abstraction functions, such as time keeping,
in ``mphalport.h``.
The following is an example of an ``mpconfigport.h`` file:
.. code-block:: c
#include <stdint.h>
// Python internal features.
#define MICROPY_ENABLE_GC (1)
#define MICROPY_HELPER_REPL (1)
#define MICROPY_ERROR_REPORTING (MICROPY_ERROR_REPORTING_TERSE)
#define MICROPY_FLOAT_IMPL (MICROPY_FLOAT_IMPL_FLOAT)
// Enable u-modules to be imported with their standard name, like sys.
#define MICROPY_MODULE_WEAK_LINKS (1)
// Fine control over Python builtins, classes, modules, etc.
#define MICROPY_PY_ASYNC_AWAIT (0)
#define MICROPY_PY_BUILTINS_SET (0)
#define MICROPY_PY_ATTRTUPLE (0)
#define MICROPY_PY_COLLECTIONS (0)
#define MICROPY_PY_MATH (0)
#define MICROPY_PY_IO (0)
#define MICROPY_PY_STRUCT (0)
// Type definitions for the specific machine.
typedef intptr_t mp_int_t; // must be pointer size
typedef uintptr_t mp_uint_t; // must be pointer size
typedef long mp_off_t;
// We need to provide a declaration/definition of alloca().
#include <alloca.h>
// Define the port's name and hardware.
#define MICROPY_HW_BOARD_NAME "example-board"
#define MICROPY_HW_MCU_NAME "unknown-cpu"
#define MP_STATE_PORT MP_STATE_VM
#define MICROPY_PORT_ROOT_POINTERS \
const char *readline_hist[8];
This configuration file contains machine-specific configurations including aspects like if different
MicroPython features should be enabled e.g. ``#define MICROPY_ENABLE_GC (1)``. Making this Setting
``(0)`` disables the feature.
Other configurations include type definitions, root pointers, board name, microcontroller name
etc.
Similarly, an minimal example ``mphalport.h`` file looks like this:
.. code-block:: c
static inline void mp_hal_set_interrupt_char(char c) {}
Support for standard input/output
---------------------------------
MicroPython requires at least a way to output characters, and to have a REPL it also
requires a way to input characters. Functions for this can be implemented in the file
``mphalport.c``, for example:
.. code-block:: c
#include <unistd.h>
#include "py/mpconfig.h"
// Receive single character, blocking until one is available.
int mp_hal_stdin_rx_chr(void) {
unsigned char c = 0;
int r = read(STDIN_FILENO, &c, 1);
(void)r;
return c;
}
// Send the string of given length.
void mp_hal_stdout_tx_strn(const char *str, mp_uint_t len) {
int r = write(STDOUT_FILENO, str, len);
(void)r;
}
These input and output functions have to be modified depending on the
specific board API. This example uses the standard input/output stream.
Building and running
--------------------
At this stage the directory of the new port should contain::
ports/example_port/
├── main.c
├── Makefile
├── mpconfigport.h
├── mphalport.c
└── mphalport.h
The port can now be built by running ``make`` (or otherwise, depending on your system).
If you are using the default compiler settings in the Makefile given above then this
will create an executable called ``build/firmware.elf`` which can be executed directly.
To get a functional REPL you may need to first configure the terminal to raw mode:
.. code-block:: bash
$ stty raw opost -echo
$ ./build/firmware.elf
That should give a MicroPython REPL. You can then run commands like:
.. code-block:: bash
MicroPython v1.13 on 2021-01-01; example-board with unknown-cpu
>>> import sys
>>> sys.implementation
('micropython', (1, 13, 0))
>>>
Use Ctrl-D to exit, and then run ``reset`` to reset the terminal.
Adding a module to the port
---------------------------
To add a custom module like ``myport``, first add the module definition in a file
``modmyport.c``:
.. code-block:: c
#include "py/runtime.h"
STATIC mp_obj_t myport_info(void) {
mp_printf(&mp_plat_print, "info about my port\n");
return mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_0(myport_info_obj, myport_info);
STATIC const mp_rom_map_elem_t myport_module_globals_table[] = {
{ MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR_myport) },
{ MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&myport_info_obj) },
};
STATIC MP_DEFINE_CONST_DICT(myport_module_globals, myport_module_globals_table);
const mp_obj_module_t myport_module = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t *)&myport_module_globals,
};
MP_REGISTER_MODULE(MP_QSTR_myport, myport_module);
You will also need to edit the Makefile to add ``modmyport.c`` to the ``SRC_C`` list, and
a new line adding the same file to ``SRC_QSTR`` (so qstrs are searched for in this new file),
like this:
.. code-block:: Makefile
SRC_C = \
main.c \
modmyport.c \
mphalport.c \
...
SRC_QSTR += modmyport.c
If all went correctly then, after rebuilding, you should be able to import the new module:
.. code-block:: bash
>>> import myport
>>> myport.info()
info about my port
>>>

25
components/language/micropython/docs/develop/publiccapi.rst Normal file
View File

@@ -0,0 +1,25 @@
.. _publiccapi:
The public C API
================
The public C-API comprises functions defined in all C header files in the ``py/``
directory. Most of the important core runtime C APIs are exposed in ``runtime.h`` and
``obj.h``.
The following is an example of public API functions from ``obj.h``:
.. code-block:: c
mp_obj_t mp_obj_new_list(size_t n, mp_obj_t *items);
mp_obj_t mp_obj_list_append(mp_obj_t self_in, mp_obj_t arg);
mp_obj_t mp_obj_list_remove(mp_obj_t self_in, mp_obj_t value);
void mp_obj_list_get(mp_obj_t self_in, size_t *len, mp_obj_t **items);
At its core, any functions and macros in header files make up the public
API and can be used to access very low-level details of MicroPython. Static
inline functions in header files are fine too, such functions will be
inlined in the code when used.
Header files in the ``ports`` directory are only exposed to the functionality
specific to a given port.

115
components/language/micropython/docs/develop/qstr.rst Normal file
View File

@@ -0,0 +1,115 @@
.. _qstr:
MicroPython string interning
============================
MicroPython uses `string interning`_ to save both RAM and ROM. This avoids
having to store duplicate copies of the same string. Primarily, this applies to
identifiers in your code, as something like a function or variable name is very
likely to appear in multiple places in the code. In MicroPython an interned
string is called a QSTR (uniQue STRing).
A QSTR value (with type ``qstr``) is a index into a linked list of QSTR pools.
QSTRs store their length and a hash of their contents for fast comparison during
the de-duplication process. All bytecode operations that work with strings use
a QSTR argument.
Compile-time QSTR generation
----------------------------
In the MicroPython C code, any strings that should be interned in the final
firmware are written as ``MP_QSTR_Foo``. At compile time this will evaluate to
a ``qstr`` value that points to the index of ``"Foo"`` in the QSTR pool.
A multi-step process in the ``Makefile`` makes this work. In summary this
process has three parts:
1. Find all ``MP_QSTR_Foo`` tokens in the code.
2. Generate a static QSTR pool containing all the string data (including lengths
and hashes).
3. Replace all ``MP_QSTR_Foo`` (via the preprocessor) with their corresponding
index.
``MP_QSTR_Foo`` tokens are searched for in two sources:
1. All files referenced in ``$(SRC_QSTR)``. This is all C code (i.e. ``py``,
``extmod``, ``ports/stm32``) but not including third-party code such as
``lib``.
2. Additional ``$(QSTR_GLOBAL_DEPENDENCIES)`` (which includes ``mpconfig*.h``).
*Note:* ``frozen_mpy.c`` (generated by mpy-tool.py) has its own QSTR generation
and pool.
Some additional strings that can't be expressed using the ``MP_QSTR_Foo`` syntax
(e.g. they contain non-alphanumeric characters) are explicitly provided in
``qstrdefs.h`` and ``qstrdefsport.h`` via the ``$(QSTR_DEFS)`` variable.
Processing happens in the following stages:
1. ``qstr.i.last`` is the concatenation of putting every single input file
through the C pre-processor. This means that any conditionally disabled code
will be removed, and macros expanded. This means we don't add strings to the
pool that won't be used in the final firmware. Because at this stage (thanks
to the ``NO_QSTR`` macro added by ``QSTR_GEN_CFLAGS``) there is no
definition for ``MP_QSTR_Foo`` it passes through this stage unaffected. This
file also includes comments from the preprocessor that include line number
information. Note that this step only uses files that have changed, which
means that ``qstr.i.last`` will only contain data from files that have
changed since the last compile.
2. ``qstr.split`` is an empty file created after running ``makeqstrdefs.py split``
on qstr.i.last. It's just used as a dependency to indicate that the step ran.
This script outputs one file per input C file, ``genhdr/qstr/...file.c.qstr``,
which contains only the matched QSTRs. Each QSTR is printed as ``Q(Foo)``.
This step is necessary to combine the existing files with the new data
generated from the incremental update in ``qstr.i.last``.
3. ``qstrdefs.collected.h`` is the output of concatenating ``genhdr/qstr/*``
using ``makeqstrdefs.py cat``. This is now the full set of ``MP_QSTR_Foo``'s
found in the code, now formatted as ``Q(Foo)``, one-per-line, with duplicates.
This file is only updated if the set of qstrs has changed. A hash of the QSTR
data is written to another file (``qstrdefs.collected.h.hash``) which allows
it to track changes across builds.
4. Generate an enumeration, each entry of which maps a ``MP_QSTR_Foo`` to it's corresponding index.
It concatenates ``qstrdefs.collected.h`` with ``qstrdefs*.h``, then it transforms
each line from ``Q(Foo)`` to ``"Q(Foo)"`` so they pass through the preprocessor
unchanged. Then the preprocessor is used to deal with any conditional
compilation in ``qstrdefs*.h``. Then the transformation is undone back to
``Q(Foo)``, and saved as ``qstrdefs.preprocessed.h``.
5. ``qstrdefs.generated.h`` is the output of ``makeqstrdata.py``. For each
``Q(Foo)`` in qstrdefs.preprocessed.h (plus some extra hard-coded ones), it outputs
``QDEF(MP_QSTR_Foo, (const byte*)"hash" "Foo")``.
Then in the main compile, two things happen with ``qstrdefs.generated.h``:
1. In qstr.h, each QDEF becomes an entry in an enum, which makes ``MP_QSTR_Foo``
available to code and equal to the index of that string in the QSTR table.
2. In qstr.c, the actual QSTR data table is generated as elements of the
``mp_qstr_const_pool->qstrs``.
.. _`string interning`: https://en.wikipedia.org/wiki/String_interning
Run-time QSTR generation
------------------------
Additional QSTR pools can be created at runtime so that strings can be added to
them. For example, the code::
foo[x] = 3
Will need to create a QSTR for the value of ``x`` so it can be used by the
"load attr" bytecode.
Also, when compiling Python code, identifiers and literals need to have QSTRs
created. Note: only literals shorter than 10 characters become QSTRs. This is
because a regular string on the heap always takes up a minimum of 16 bytes (one
GC block), whereas QSTRs allow them to be packed more efficiently into the pool.
QSTR pools (and the underlying "chunks" that store the string data) are allocated
on-demand on the heap with a minimum size.

70
components/language/micropython/docs/develop/writingtests.rst Normal file
View File

@@ -0,0 +1,70 @@
.. _writingtests:
Writing tests
=============
Tests in MicroPython are located at the path ``tests/``. The following is a listing of
key directories and the run-tests.py runner script:
.. code-block:: bash
.
├── basics
├── extmod
├── float
├── micropython
├── run-tests.py
...
There are subfolders maintained to categorize the tests. Add a test by creating a new file in one of the
existing folders or in a new folder. It's also possible to make custom tests outside this tests folder,
which would be recommended for a custom port.
For example, add the following code in a file ``print.py`` in the ``tests/unix/`` subdirectory:
.. code-block:: python
def print_one():
print(1)
print_one()
If you run your tests, this test should appear in the test output:
.. code-block:: bash
$ cd ports/unix
$ make tests
skip unix/extra_coverage.py
pass unix/ffi_callback.py
pass unix/ffi_float.py
pass unix/ffi_float2.py
pass unix/print.py
pass unix/time.py
pass unix/time2.py
Tests are run by comparing the output from the test target against the output from CPython.
So any test should use print statements to indicate test results.
For tests that can't be compared to CPython (i.e. micropython-specific functionality),
you can provide a ``.py.exp`` file which will be used as the truth for comparison.
The other way to run tests, which is useful when running on targets other than the Unix port, is:
.. code-block:: bash
$ cd tests
$ ./run-tests.py
Then to run on a board:
.. code-block:: bash
$ ./run-tests.py --target minimal --device /dev/ttyACM0
And to run only a certain set of tests (eg a directory):
.. code-block:: bash
$ ./run-tests.py -d basics
$ ./run-tests.py float/builtin*.py