[feature preview] User-defined operators

Feedback
#1

We’re aiming to release solc 0.8.18 somewhere next week and one of the major new features will be the support for user-defined operators. We usually only get feedback on features after they are released so I thought I’d try something different for a change. Below are the links to the dev binaries from our CI with the feature implemented. Feel free to try them out (version 0.8.18-ci.2023.1.17+commit.e7b959af):

You can read a detailed description of how the feature works. It’s up to date except for one thing that was changed after it was written: operators can now only be implemented with a pure function.

I invite everyone to try them out and comment on what you like or dislike about the current design and implementation. Here are some questions that I’d be particularly interested in but any feedback is welcome.

  • Should we allow using non-pure functions for operators?
  • Given current limitations, do you think operators on structs would still be useful?
  • ! returns the same type it takes rather than bool. Should it be returning bool instead?
  • Are the error messages clear and detailed enough?

UPDATE 2023-02-09
Here’s a new preview (version 0.8.19-ci.2023.2.8+commit.0cf7cd30):

Changes:

  • It’s no longer possible to define <<, >>, ** and !
  • Operator definition must be a free function. Library functions (especially external ones) are no longer allowed.
  • Operator definitions must be global. It’s not possible to define an operator in contract scope or for UDVTs defined in contract scope.
  • Multiple conflicting definitions are detected immediately in using for rather than when the operator is actually used.
  • SMTChecker now warns that user-defined operators are not yet properly supported.
3 Likes
#2

Any way to actually connect through a Hard-Hat testing environment?
In other words, what should I write in my hardhat.config.js file, as version: "0.8.18" is obviously not gonna work?
Thanks

#3

You have to create a custom task to run a custom solc binary. See hardhat-examples/custom-solc.

That’s unfortunately a bit more complicated than just putting in the path but they also have a feature request for that: User should be able to set solc compiler path to be used by hardhat · Issue #3294 · NomicFoundation/hardhat · GitHub so it may become easier in the future.

#4

Well, I have a BigNumber contract (see here), which IMO is perfectly suitable for testing this new feature with, as it implements all the arithmetic operators (as functions of course).

But the the preliminary setup requirements sound a bit deterring.

#5

Unfortunately there’s not much we can do about that. We’ll have to wait for that feature request to be implemented on their side.

In the meantime, you can easily run the emscripten binary with Remix. Not sure how easy it is to make your project run on that but you can at least play with the feature that way.

#6

Well, I’d imagine that validating this new solc feature would be best via massive (automated) testing, an infrastructure which is already well established in my project.

I was hoping to upgrade my contracts and rerun the test suite with zero-to-minimal changes (though I’d probably need to by the least change these tests to call the contract’s operators instead of functions).

So I’m not sure how good a verification I can achieve via Remix.

FYI:

I suppose that this new feature is intended mostly for internal functionality, but I wonder what kind of syntax would allow calling such functions (operators) directly from a JS or TS script.

I cannot quite see how the syntax for that would remain as clean as the operators in Solidity are, which is likely to deter a lot of the community (everyone other than those who use Python for web3 interaction).

Thanks

#7

Well, I’d imagine that validating this new solc feature would be best via massive (automated) testing, an infrastructure which is already well established in my project.

Sure that would be very helpful. It was pretty extensively tested on our side but more testing does not hurt. Especially with a real-life project.

I was hoping to upgrade my contracts and rerun the test suite with zero-to-minimal changes (though I’d probably need to by the least change these tests to call the contract’s operators instead of functions).

I mean, it’s not like using a custom binary with Hardhat is that complicated. Look at the template of hardhat.config.js. It’s just a few lines to paste into your config and adjust variables used in it. You can also look at how we do it for our own testing, which works with TypeScript as well.

It’s not as simple as changing one setting so might be an issue for a casual user or be a bit tedious if you just want to quickly try it out but it’s still relatively simple if you’re an advanced user. It’s still just a config change, not a change in Hardhat’s source. Looks like you’re already going to put quite a bit more effort into this so I don’t think this would be an obstacle for you.

I suppose that this new feature is intended mostly for internal functionality, but I wonder what kind of syntax would allow calling such functions (operators) directly from a JS or TS script.

Do you mean calling operators via an external call from the outside of a contract? What would be the use case for that? There are currently no plans to make that possible.

#8

That template looks like exactly what I need.
I’ll see if I can get it working quickly.
Tx.

#9

My suggestion would be to only allow the not operator for user defined types whose underlying type is bool. Otherwise, the not operator will just be used to check an arbitrary condition on a value type and no longer mean logical negation. I think enforcing the latter option would be best for clarity and be in line with the existing semantics of the not operator.

Consider the following code:

type Int is int128;
using {
    not as !
} for Int;

function not(Int x) pure returns (bool) {
    return Int.unwrap(x) == 0;
}

function a() pure returns (bool) {
      require(!Int.wrap(1));
      if (!Int.wrap(1)) {
          return false;
      }
      return !Int.wrap(1);
  }
// function does_not_compile() pure returns (bool) {
//       require(!int128(1));
//       if (!int128(1)) {
//           return false;
//       }
//       return !int128(1);
//   }

If I were to try this with the underlying type, int128, I would receive a type error indicating that the unary not operator is not available for this type. I don’t see why wrapping it in a type alias should change that behavior. What the function not is doing is arbitrary and isn’t related to logical negation as that operation cannot be performed on it’s underlying type. It would make since to instead define a helper function instead:

function is_zero(Int x) pure returns (bool) {
     return Int.unwrap(x) == 0;
}
function b() pure returns (bool) {
      require(is_zero(Int.wrap(1)));
      if (is_zero(Int.wrap(1))) {
          return false;
      }
      return is_zero(Int.wrap(1));
}

(cross posted from the discussion on the PR User-defined operators for UDVTs by cameel · Pull Request #13790 · ethereum/solidity · GitHub)

#11

@cameel: OK, got it working!

Now, following your example, I have added this at the top of my contract:

pragma solidity ^0.8.17;

import "../NaturalNum.sol";

using {NaturalNum.add as +} for uint256[];

Where the actual contract is:

contract NaturalNumUser {
    function add(
        uint256[] memory x,
        uint256[] memory y
    ) external pure returns (uint256[] memory) {
        return NaturalNum.add(x, y);
    }
}

I’d like to replace return NaturalNum.add(x, y) with return x + y of course.

But before even trying that, the compiler doesn’t seem to accept the new syntax:

DeclarationError: Identifier not found or not unique.
 --> project/contracts/helpers/NaturalNumUser.sol:6:8:
  |
6 | using {NaturalNum.add as +} for uint256[];
  |        ^^^^^^^^^^^^^^


Error HH600: Compilation failed

And if I change pragma solidity ^0.8.17 to pragma solidity 0.8.18, then I get:

ParserError: Source file requires different compiler version
(current compiler is 0.8.18-ci.2023.1.17+commit.e7b959af.Darwin.appleclang) -
note that nightly builds are considered to be strictly less than the released version
 --> project/contracts/NaturalNum.sol:2:1:
  |
2 | pragma solidity 0.8.18;
  | ^^^^^^^^^^^^^^^^^^^^^^^

Any suggestions?

UPDATE:

The reason for the Identifier not found or not unique error, believe it or not, is the fact that I have an overloaded private function of the same name, i.e.:

function add(...) private pure ...

So this definitely sounds like something that you need to fix in the compiler.

Trying “using {NaturalNum.eq as ==} for uint256” instead, I get a different compilation error:

Operators can only be implemented for user-defined value types

This is because the compiler insists that I first define the specific type, for example:

type xxx is uint256;
using {NaturalNum.eq as ==} for xxx;

It is not really clear to me why this requirement is needed in the first place (i.e., why not just allow using uint256 as part of the using statement?).

But in any case, my type happens to be uint256[], which the compiler doesn’t even support (“The underlying type for a user defined value type has to be an elementary value type”).
So in short, the new feature is not yet “ripe” for me to make use of.

BTW, it would be nice if you could add support for non-elementary user defined value types at some point (see this issue, which I have opened on your GitHub some time ago).

Thanks :slight_smile:

#12

I worry that this would make security auditors’ jobs harder, and also encourage gas inefficient abstractions. Solidity has historically been quite strict and this seems like a divergence from that.

Is there any plan to add warnings related to this feature to solc?

Operators can only be defined for user-defined value types for now.

This “for now” part is what makes me concerned. Overloading a built in operator for built in types should be disallowed or warned against.

#13

Regarding the JS compiler: it’s the emscripten binary. Emscripten compiles C++ code compiled to wasm and adds a small JS wrapper. It can be loaded from JS code.

But you can use native binaries just fine with Hardhat. You just have to set isSolcJs to false.

But before even trying that, the compiler doesn’t seem to accept the new syntax:

That’s a bug and it’s not specific to operators.

external functions cannot be currently attached with using for. We’re working on a fix for this but it unfortunately will probably not make it into 0.8.18.

You can still attach public functions though. In your example you probably want a public function anyway because this will allow you to call it within the library internally, while still having the call external outside of it.

And if I change pragma solidity ^0.8.17 to pragma solidity 0.8.18, then I get:

>0.8.17 is what you want here. These compiler binares have a pre-release version, which matches neither 0.8.17 nor 0.8.18. It’s treated as something in between for the purpose of the pragma.

The reason for the Identifier not found or not unique error, believe it or not, is the fact that I have an overloaded private function of the same name, i.e.:

That’s a limitation of overloaded functions in Solidity in general, not just operators. You’ll get that with plain using for as well. Or when using function pointers. There’s no way to specify which overload you want so it’s ambiguous.

This will be fixed eventually but for now there are bigger priorities so it has to wait.

Trying “using {NaturalNum.eq as ==} for uint256” instead, I get a different compilation error:

That’s by design. User-defined operators are currently only meant to work with user-defined value types. We want to make them more ergonomic and as convenient as built-in value types, not to allow redefining behavior of built-in types. We might eventually allow them for structs or other named types but for value types it’s very unlikely. If your integer is supposed to behave differently from a normal integer, you should define a new type for it.

But in any case, my type happens to be uint256[], which the compiler doesn’t even support (“The underlying type for a user defined value type has to be an elementary value type”).

Some form of typedefs may come to the language eventually, though for now we have no design decision on that. For now the only way to work around it is to wrap your type in a struct.

This case is an argument for having operators on structs as well. Noted. For now though we decided not to do reference types because we’re planning changes in how locations work and these changes would be likely breaking for these operators.

#14

I worry that this would make security auditors’ jobs harder

I think it’s a valid concern, which is why this feature is limited to cases where you could not define operators before. And why by default operators are only available in the same module (you have to explicitly use using for global to override that).

There’s definitely some potential for abuse but it’s a trade-off and I think that specifically for user-defined value types is a net positive. The intention is to make them more usable, on par with built-in types, especially for use cases like fixed-point numbers. UDVTs let you write more type-safe code which eliminates a whole class of potential mistakes.

Is there any plan to add warnings related to this feature to solc?

What kind of warnings would you like to see?

This “for now” part is what makes me concerned. Overloading a built in operator for built in types should be disallowed or warned against.

It is disallowed and there are no plans to allow it. “for now” refers to the fact that eventually we’d like structs to be supported as well. Maybe enums, though not sure how useful that would be. Can’t say for use that other reference types won’t ever be supported via some kind of typedef feature but those do not have built-in operators anyway.

#15

My suggestion would be to only allow the not operator for user defined types whose underlying type is bool.

We discussed this briefly in the team and decided to remove support for ! from the current PR. We’ll likely allow it eventually but for now opinions were divided so we need more time to make a decision.

One thing we’re not too keen on though is allowing it only for specific underlying types like bool. If we add it, it will likely be for any user-defined value type.

#16

I have considered it, but the result is an increase in both code size (deploy-time gas cost) and run-time gas cost.

Tx.

#17

I’m sorry, perhaps I am missing something, but this new feature (operator-overloading) seems pretty useless when one can apply it only on built-in types, since most of these types already support the usage of operators to begin with.

#18

Unfortunately it looks like the feature as it is now does not fit your NaturalNum too well because it’s a dynamic reference type. I wouldn’t call it useless just because of that. We’ll consider supporting reference types better in the future but our approach is to tread carefully and start with something simple that we’re certain about. There were already concerns about auditability in this thread and we don’t want the mechanism to end up being as elaborate as in some other languages (namely, C++). But we do intend to extend it in ways that improve certain use cases. For now these were: creating more semantic types via UDVTs and implementing new elementary types on top of existing value types (e.g. fixed-point numbers).

#19

I apologize if I was misunderstood, I did not call it useless just because it doesn’t fit my specific case.

I called it useless because it can be applied only on built-in types, most of which already support the usage of arithmetic operators to begin with.

So other than implementing a library for stuff like concatenating strings in a natural manner, I cannot quite see what else it can be useful for.

Typically, overloading arithmetic operators becomes a useful feature when applied on complex objects.

I’m aware of the auditability concern, but this argument is orthogonal to the discussion here IMO, as it would apply for pretty much every other new feature added to the language.

Thanks again :slight_smile:

#20

You cannot apply them to built-in types (like uint or address). Only to user-defined value types, which currently have no operators. The only thing you could really do so far with UDVTs was to convert to/from built-in types with wrap()/unwrap().

UDVTs were meant to be a way to add more semantic information in situations where you currently use built-in value types and operators are a step towards that. This is similar to the current distinction between address and address payable. Both store the same exact value but have different constraints. With UDVTs you can implement your own types of this kind. For example you might replace something like buy(uint when, uint price) with buy(Timestamp when, DAIValue price) to make sure that timestamps and prices cannot be mixed up. This can eliminate a whole class of bugs. Operators are crucial here because without them you cannot operate on these types directly. You have to unwrap them, by which you lose type safety.

Another use case is implementing new elementary value types that only use the built-in ones for the underlying representation and do not share their semantics. A good example is a fixed-point type. A simple decimal fixed-point type can be implemented by storing the value multiplied by a power of 10 in an uint. Many operations on uints can destroy the value though if used in a naive way. For example the multiplication of two such numbers as uints will not give you the right result. You can work around this by wrapping these operations in helper functions but this does not remove the built-in operator and is not nearly as safe as creating a dedicated UDVT and having a custom * operator that does the right thing.

#21

I’m sorry, but unless I’m missing something here, you’re just arguing semantics, since user-defined value types are currently supported only for built-in types.

For example, this is supported:

type MyType is uint256;

While this is not supported:

struct Uint512 {
    uint256 hi;
    uint256 lo;
}
type MyType is Uint512;

And so the only types which I can implement arithmetic operators for, are user-defined value types which directly correspond to built-in types, such as uint256, string, address, etc.

How exactly is that useful, when I can simply apply arithmetic operators on the (original) built-in type?

True - it won’t work for strings and addresses, but I find this to be of very little value for these specific types (while supporting arithmetic operations between a pair of Uint512 operands, for example, would be of a much higher value).

Thanks