I wish to undertake a very large, long-term project for a web application/website. It will be something like a social networking/video sharing. Think twitter/tiktok
I’d like to apologize if this is not the right place to post this question or if it has been asked before. I’ve looked through many threads if a question similar to mine has been asked, but non quite in-depth enough.
My question relates to what tools and topics do I need to learn about to accomplish my goals?
I have some android and sqlite/realmDB development experience but am novice when it comes to web development. Though not afraid to get my hands dirty and put in the time. Because of this, I am looking for more technical responses or suggestions for where I can find more about it.
Efficiency, security and modularity are are essential to me.
Front-end - ? The look is the last thing on my mind, but any tips for the css/html? Like plugins or extensions to create dynamic, responsive pages. Is there there alternatives to JavaScript? Or at least workarounds for it’s potential of malicious exploitation.
DBMS - I assume relational is the correct approach. Any recommended DB’s? Is there a place where I can find examples of data models, say for example, youtube or facebook? Analytics are also important
Back-end/Virtualization - What web sever software? (apache, oracle…etc)
I’m thinking C/C++ but am also considering Erlang for its efficiency
I want to go with containers using docker, rather than virtualization hypervisors , please tell me if there’s a better way
micro-services - How would I even begin to approach this? REST seems to be important to learn, but how many more/what API’s would I need to implement?
Really, thank you for reading that whole thing! ANY response for ANY section is greatly appreciated,
I’d start by thinking about the whole thing as a pipeline. There is some information you have in a database you want to send to the front end (web browser or app) or you have some information in the front end you want to store in the database. There is something that sits between your cue/angular/react/plain JavaScript website and your database. That is probably the thing you want to run in containers.
This is also a good time to introduce security concepts. The thing you run in containers should not trust the front end at all. Expose the bare minimum you need for the front end to work. Log everything.
Consider a prebuilt solution like Hasura if you’re planning to go with graph ql.
The suggestion to think of it as a pipeline was REALLY helpful. Prior It was just a bunch of different modules in my head with no connection in between. You’ve also given me another lead to follow in terms of security, thank you or that. I will look into Hasura and graphql
MongoDB, JSON, SQLite are good for this kind of approach. You’'ll need a ETL process for the data stream and tie that all up with the pipeline concept. I worked in a place where we had a Tablet for end users utilizing JSON to save into a MongoDB database, and we created a Kinesis data stream for the ETL process to dump the data into a Redshift Database(SQL) and an XML dump for a mobile app being developed.
Another great suggestion, an ETL process. This really helps get the ball rolling. Creating a pipeline will be one of the firsts tasks I undertake after I finish the research stage.
I feel it will be a while before I can start looking at what DBMS to with as there are many considerations I need to take into account (like integration with other services) and will mostly depend on what kind of pipeline I develop. I am leaning more towards distributed noSQL DB’s because of the scale and concurrency I’m planning
There is webassembly I guess. You can look into blazor. I dont know any other frameworks that use web assembly.
But honestly javascript being ‘exploitable’ is a bad excuse of saying my backend is insecure af and cant handle wrong user inputs or gives away information it really should not in the first place.
If thats the case its exploitable either way. You just also hand out code samples so its easier to figure out how your backend works by looking at the js code. Security by obscurity isnt really a good basis to base your security on. You should think of your js client code as being basically an openbook / open source. Dont put things in there the user should not have access too.
I would not recommend you go with c/c++. Just takes a lot of time…
I would recommend nodejs or .net core for restapis (though there are many other options in that space). JWT tokens are pretty good for stateless authentication. Ultimately in a restapi you want to never keep any state what so ever. This is so your restapi can be duplicated without having it break when a user gets load balanced to a different restapi in the middle of using your program.
About deployment you dont really initially have to worry about any of that. Both .net core and nodejs come with their own webserver. Though it is recommended for production use to put it behind an reverse proxy. NGINX would be good for that.
In both nodejs and .net core creating docker containers is basically free bread. So it is pretty simple to get a container image of your program if you later want to use kubernetes for container orchestration.
Imo, either one of the big js frameworks would be best. Vue, react and angular. Alternatively you can use blazor. Though im not toally sure how good it is yet. I would recommend it over asp.net any day and it is the only fully webassembly based framework I know of. For the most part webassembly is more considered to build parts of an application with c++ with it that needs the performance not to replace your entier front end stack with. But blazor seems to try exactly that.
For the start you could also gobble together some site with just html and js to test things with.
MongoDB is pretty good. But there is nothing wrong with postgresql. Oracle and mssql are fine too. They just also cost you. And I dont think you get much for your money over postgresql other than enterprise support. In relational databases you can do more complex queries with joins and stuff. They also have far better concepts to ensure data integrety. But mongodb can often outperform a rdms and can be a much simpler db to manage.
You can also use redis for response caching. If you did a good job following restful architecture. It should be fairly easy to do response caching for your get requests. For instance you can cache the content of your landing page for 5 minutes and hit your database only once every 5 minutes instead of every single request. You dont want to keep variables for caching in your restapi code. As then you introduce state into it. Thats why you should use redis if you wanna do that.
Your videos you do not want to store in your database. Its easiest if you just store them on disk and keep the path to it in your database.
What’s your intention with this project? To get a job or just for funsies?
Cause if you want a job you should look on your region what frameworks people are using and go with them.
If you’re on the USA chances are: C# for backend, probably sql server with some azure or something else with aws and either Angular, Vue or React as a front-end.
People want to be able to build fast instead of having light applications that take around a year to build.
If you just in for the lolz then use C for backend and pure js with html and css, you’d take a looong time doing it. Also you’d need either bash or ps to deploy it.
Frontend: i would suggest picking a tool with a good comunity (big and helpfull) like react, it will make everything a lot easier.
DB: this will depends heavely on what your doing, there are several choices now day that are prod ready and can scale significantly, depending on your data needs.
Backend: C/C++ for a web backend is not very common, erlang isent either, velocity i usually appreciated more than performance, most companies start with a faster language, nodejs, python, golang once you know where your performance requires improvements you can work towards that with languages that adapt to that workflow. containers is the way to go now days.
micro services: this is a huge world, microservices are not easy, there are a new set of problems and and increase in complexity, starting anything with microservices is the wrong aproach as you are not solving any problem just creating more. also if you have different components talking toguether avoid rest and use something like GRPC. leave rest as layer to talk to your frontend.
there are so many tools and frameworks out there that fit spesifict problems that without knowing what you are trying to solve its hard to recomend.
I would prefer to start with a solid foundation from the get-go. It might mean that results will take longer to produce, but wouldn’t that be better than having to recode things in another language?
Integration of microservices is more of a long-term consideration with the project and won’t actually implement it until I have something more concrete. With that being said, I’d like a base that would be easily configure and adaptable. For the API, I’ve heard only things about REST, your suggestion of GRPC I will definitely consider going forward
A more detailed description of what I want the project to be is a social media and video sharing platform. Like a combination of youtube and facebook. I’ve looked around as to what stack the top companies use, facebook has erlang in the back-end which I thought was interesting since no large company has ever rolled it out. It might’ve ebeen an understatement to say it was a large project
Such a detailed response, awesome! You’ve given much to think about.
JavaScript is unavoidable it seems. But quite it’s versatile and its capabilities really depends on the programmer. No complaints anymore with it.
unless you have the scale of facebook|google|amazon thinking about high performance language is an error, none of them are good general programing languages and it will slow you down. its generally better to have something slow than to have nothing. its to arrogant to think you know where performance hit will exist. my experience after 10+ years in the bay area working for several startups (most of them pretty succesfull) is that bottlenecks will always surprise you.
so pick something fast that will allow you to get a product fast and add metrics and performance metrics everywhere, that will allow you to identify bottlenecks and decide based on information how to act (either refactor in the same language or write a module for that language for that bottleneck in a more performant language like C)
social structures are not very SQL friendly you may be better with a DB that works well with graph structures couchbase if a good example of that, but there are quite a few of them.
given social data like youtube or facebook it may be of interest to you to work with graphql api for your frontend.
Yes and they also have python, java, php and c++. They are giant. They started out with just php though. Microsoft has a guy who works on pretty much just the profile page(s) all day every day for the entier year. You cant really decide your tech stack based on any of that to start out with.
Starting out with php would be fine today. Then you look at twitter who started with ruby. Witch if they started again today, they would probably pick something other than ruby.
You might be better off with 5 projects instead of one using fast-to-deploy frameworks… you’d have 5 times more projects on your portfolio…
You might also want to consider node.js, it’s pretty nice. Javascript can be confusing at times but it’s nice to code.
Search which front-end framework fits your needs and get to coding. Might eventually build mobile apps to go with them also, in this case flutter might be good if you don’t want to go native, if native: swift for ios and kotlin for android (or java if you like to suffer)
Could elaborate more? I don’t quite get what you mean
@lacion
Really thanks for that advice. Your outlook really puts things into perspective for me. Getting something out first is more important than making sure its perfect before you even see anything
@anon85095355
That’s sort of what I had been going for. A modular approach/pipeline. Some hints would be appreciated
This is more from a business development point of view: you want to build something fairly quickly and is fairly easy to read so you can decide whether to spend tons of money or time on it or not. Just what I’ve read online. For me, I think this stack is postgresql, flask (python 3), and maybe angular.
I am full stack well developer and here are my 2 cents.
I would go with python Django or flask for backend. Django now got ASGI (async) support in 3.0 version. As people at google said "python where whe can to C where we have to. ’
I would go with React for front end.
As project grows if needed i would add C code for services that needs to run faster.
other alternative is Go, its pretty fast and stable.
